Jul 16 21:45:22.953636 ip-10-0-143-124 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:45:22.953648 ip-10-0-143-124 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:45:22.953658 ip-10-0-143-124 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:45:22.953961 ip-10-0-143-124 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:45:32.691832 ip-10-0-143-124 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:45:32.691846 ip-10-0-143-124 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 15ea4435d1664fe3a6ba30f871a04c4c --
Jul 16 21:47:22.841357 ip-10-0-143-124 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:47:23.214236 ip-10-0-143-124 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:47:23.214236 ip-10-0-143-124 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:47:23.214236 ip-10-0-143-124 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:47:23.214236 ip-10-0-143-124 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:47:23.214236 ip-10-0-143-124 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:47:23.215117 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.214986    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:47:23.219896 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219880    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:23.219896 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219896    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219899    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219903    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219906    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219909    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219912    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219915    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219919    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219921    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219924    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219927    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219930    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219932    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219935    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219938    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219941    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219944    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219947    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219949    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:23.219954 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219952    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219956    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219959    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219961    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219964    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219967    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219971    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219974    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219976    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219980    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219983    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219985    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219988    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219990    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219993    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219996    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.219998    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220001    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220004    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220006    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:23.220414 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220009    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220011    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220014    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220017    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220019    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220023    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220026    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220028    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220031    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220033    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220036    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220038    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220041    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220044    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220047    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220050    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220052    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220055    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220058    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220060    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:23.220919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220063    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220065    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220068    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220073    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220077    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220079    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220096    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220099    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220101    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220104    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220106    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220110    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220112    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220115    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220118    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220120    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220123    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220126    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220128    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:23.221397 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220133    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220137    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220140    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220144    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220148    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220151    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220154    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220159    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220162    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220165    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220168    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220171    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220174    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220177    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220180    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220183    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220186    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220189    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220192    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220194    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:23.221848 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220886    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220893    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220896    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220899    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220903    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220906    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220909    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220912    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220914    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220917    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220920    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220923    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220926    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220929    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220931    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220934    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220937    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220939    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220942    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220945    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:23.222323 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220947    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220950    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220952    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220955    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220957    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220960    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220962    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220965    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220968    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220970    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220973    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220975    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220978    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220980    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220985    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220988    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220992    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220995    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.220998    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:23.222813 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221001    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221004    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221006    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221009    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221012    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221014    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221017    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221019    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221022    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221025    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221027    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221031    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221034    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221037    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221039    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221042    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221044    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221047    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221050    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221052    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:23.223282 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221056    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221060    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221063    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221066    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221068    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221071    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221073    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221076    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221078    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221097    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221101    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221103    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221111    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221114    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221116    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221121    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221124    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221127    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221129    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221132    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:23.223815 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221135    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221137    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221140    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221142    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221145    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221148    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221150    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221153    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221156    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221158    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221161    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221163    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221166    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221169    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221171    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221173    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221176    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221178    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221181    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221183    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:23.224325 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221554    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221559    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221561    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221564    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221568    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221571    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221573    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221576    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221579    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221582    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221584    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221587    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221590    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221592    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221594    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221597    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221600    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221602    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221605    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221607    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:23.224788 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221609    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221612    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221615    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221617    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221621    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221625    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221628    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221631    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221634    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221637    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221639    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221642    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221645    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221648    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221650    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221652    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221655    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221658    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221660    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221664    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:23.225284 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221666    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221669    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221671    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221674    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221676    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221679    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221681    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221684    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221686    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221689    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221692    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221694    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221696    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221699    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221702    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221704    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221708    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221710    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221713    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:23.225755 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221715    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221718    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221720    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221723    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221725    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221728    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221731    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221733    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221736    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221739    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221743    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221746    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221749    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221752    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221754    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221757    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221759    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221762    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221764    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:23.226217 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221767    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221770    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221772    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221775    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221777    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221780    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221782    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221785    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221787    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221790    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221792    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221796    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221798    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221801    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221804    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221806    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221808    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221811    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221814    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221818    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:23.226672 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.221821    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221895    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221910    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221916    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221921    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221926    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221929    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221933    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221937    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221940    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221943    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221947    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221950    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221953    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221956    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221959    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221962    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221965    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221967    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221971    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221974    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221977    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221980    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221983    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:47:23.227269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221987    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221990    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221993    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.221996    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222001    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222004    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222008    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222011    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222014    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222018    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222020    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222023    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222026    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222029    2555 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222032    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222037    2555 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222040    2555 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222043    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222045    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222048    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222052    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222055    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222058    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222061    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222064    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:47:23.227848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222067    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222069    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222072    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222076    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222078    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222094    2555 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222098    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222101    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222104    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222107    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222110    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222115    2555 flags.go:64] FLAG: --help="false"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222119    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222122    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222125    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222128    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222131    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222135    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222138    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222141    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222143    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222147    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222151    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222154    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:47:23.228536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222157    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222160    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222163    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222166    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222169    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222172    2555 flags.go:64] FLAG: --lock-file=""
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222174    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222177    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222180    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222185    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222188    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222191    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222193    2555 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222196    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222199    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222202    2555 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222205    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222209    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222212    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222217    2555 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222224    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222227    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222230    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222233    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222236    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:47:23.229121 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222239    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222242    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222249    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222252    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222255    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222258    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222262    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222267    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222270    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222273    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222277    2555 flags.go:64] FLAG: --port="10250"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222280    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222282    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f7796eca92ccdd9"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222286    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222289    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222291    2555 flags.go:64] FLAG: --register-node="true"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222294    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222300    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222303    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222306    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222308    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222312    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222315    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222318    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222321    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:47:23.229726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222324    2555 flags.go:64] FLAG: --runonce="false"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222327    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222333    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222336    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222339    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222342    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222345    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222349    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222352    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222355    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222358    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222360    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222363    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222366    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222369    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222372    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222377    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222380    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222383    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222387    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222390    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222392    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222395    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222398    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222401    2555 flags.go:64] FLAG: --v="2"
Jul 16 21:47:23.230356 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222404    2555 flags.go:64] FLAG: --version="false"
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222408    2555 flags.go:64] FLAG: --vmodule=""
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222412    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.222415    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222562    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222577    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222581    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222585    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222588    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222591    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222594    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222597    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222600    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222603    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222606    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222609    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222612    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222615    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222618    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222621    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222623    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:23.230964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222626    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222629    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222632    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222635    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222637    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222640    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222643    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222645    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222648    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222650    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222653    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222655    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222659    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222663    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222666    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222669    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222673    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222676    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222678    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222681    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:23.231497 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222685    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222687    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222690    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222693    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222695    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222698    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222701    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222703    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222706    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222708    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222712    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222717    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222720    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222723    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222726    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222729    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222731    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222734    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222736    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:23.231996 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222739    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222742    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222744    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222747    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222749    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222752    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222755    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222757    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222760    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222764    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222767    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222770    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222772    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222775    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222778    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222781    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222785    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222788    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222791    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:23.232473 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222794    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222797    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222800    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222803    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222805    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222808    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222810    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222813    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222816    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222818    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222821    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222824    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222827    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222830    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222832    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222835    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222838    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222840    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222843    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222845    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:23.232970 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222848    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222851    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222854    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.222857    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.223485    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.230345    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.230369    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.231057    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.231068    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231159    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231166    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231169    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231173    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231176    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:23.233465 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231179    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231182    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231185    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231188    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231191    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231194    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231196    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231199    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231202    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231204    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231207    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231210    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231212    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231215    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231218    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231220    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231223    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231225    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231228    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231231    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231233    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:23.233812 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231236    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231239    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231241    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231254    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231258    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231262    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231265    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231267    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231270    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231273    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231277    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231280    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231283    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231286    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231289    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231292    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231295    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231298    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231300    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:23.234406 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231303    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231305    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231308    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231310    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231313    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231315    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231318    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231320    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231323    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231326    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231328    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231331    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231333    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231336    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231345    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231348    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231351    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231360    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231362    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231365    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:23.234855 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231368    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231370    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231373    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231376    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231378    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231381    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231383    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231386    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231388    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231391    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231395    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231398    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231401    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231403    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231406    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231408    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231411    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231413    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231416    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231418    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:23.235343 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231420    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231423    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231425    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231428    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231431    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231433    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231435    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231438    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231442    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231444    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231453    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231455    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231458    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231460    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.231466    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:47:23.235835 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231641    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231647    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231650    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231653    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231656    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231658    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231661    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231663    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231666    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231668    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231671    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231673    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231676    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231678    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231681    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231683    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231686    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231688    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231691    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231694    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:23.236259 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231696    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231699    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231701    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231705    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231709    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231712    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231716    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231720    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231729    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231733    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231735    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231738    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231740    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231743    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231745    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231748    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231760    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231763    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231765    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:23.236732 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231768    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231770    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231773    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231775    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231777    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231780    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231784    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231787    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231790    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231793    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231796    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231799    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231802    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231805    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231807    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231810    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231812    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231815    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231818    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231821    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:23.237208 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231823    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231826    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231835    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231838    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231840    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231843    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231846    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231848    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231851    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231853    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231856    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231858    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231861    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231863    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231866    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231868    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231871    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231874    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231876    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231878    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:23.237704 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231881    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231883    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231886    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231889    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231891    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231894    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231896    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231899    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231901    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231904    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231906    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231909    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231912    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231915    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231917    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231925    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231928    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231930    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231933    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:23.238257 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:23.231935    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:23.238701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.231941    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:47:23.238701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.232065    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:47:23.238701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.234032    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:47:23.238701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.235223    2555 server.go:1019] "Starting client certificate rotation"
Jul 16 21:47:23.238701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.235315    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:47:23.238701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.236074    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:47:23.238701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.237439    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:47:23.244063 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.244040    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:47:23.256621 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.256603    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:47:23.259145 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.259128    2555 log.go:25] "Validated CRI v1 image API"
Jul 16 21:47:23.259937 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.259920    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:47:23.262030 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.262003    2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a4a9227-5190-4180-8dbd-bfff65b16a05:/dev/nvme0n1p4 c01384b3-cb2a-4963-9a0f-15ec9f23381d:/dev/nvme0n1p3]
Jul 16 21:47:23.262129 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.262027    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:47:23.266905 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.266784    2555 manager.go:216] Machine: {Timestamp:2026-07-16 21:47:23.265704183 +0000 UTC m=+0.312739762 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100255 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2938b09ac79e66b99cb6a16ecd314b SystemUUID:ec2938b0-9ac7-9e66-b99c-b6a16ecd314b BootID:15ea4435-d166-4fe3-a6ba-30f871a04c4c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:34:ba:3e:2f:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:34:ba:3e:2f:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:63:b9:73:33:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:47:23.266905 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.266901    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:47:23.267053 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.266984    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:47:23.267909 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.267882    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:47:23.268041 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.267912    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:47:23.269001 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.268991    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:47:23.269034 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.269005    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:47:23.269034 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.269012    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:47:23.269599 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.269586    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:47:23.269746 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.269731    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:47:23.270623 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.270613    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:47:23.274189 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274177    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:47:23.274302 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274293    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:47:23.274346 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274321    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:47:23.274346 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274331    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:47:23.274346 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274345    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:47:23.274457 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274354    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:47:23.274457 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274363    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:47:23.274706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274693    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:47:23.274750 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.274719    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:47:23.276797 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.276781    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:47:23.278035 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278022    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:47:23.278075 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278052    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:47:23.278905 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278894    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:47:23.278950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278909    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:47:23.278950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278915    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:47:23.278950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278920    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:47:23.278950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278929    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:47:23.278950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278936    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:47:23.278950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278945    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:47:23.278950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278950    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:47:23.279157 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278956    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:47:23.279157 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278962    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:47:23.279157 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278975    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:47:23.279157 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.278981    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:47:23.280065 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.280054    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:47:23.280065 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.280065    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:47:23.283268 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.283253    2555 server.go:1268] "Started kubelet"
Jul 16 21:47:23.283385 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.283337    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:47:23.283449 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.283422    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:47:23.283449 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.283438    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:47:23.284142 ip-10-0-143-124 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:47:23.285165 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.285129    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:47:23.288361 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.288337    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:47:23.288522 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.288497    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:47:23.288581 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.288506    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:47:23.289192 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.289170    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:47:23.290138 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.290102    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:47:23.293400 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.293384    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:47:23.293481 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.293443    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:47:23.293532 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.293485    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.294119 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294099    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:47:23.294119 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294097    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:47:23.294269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294127    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:47:23.294269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294223    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:47:23.294269 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294234    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:47:23.294714 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294700    2555 factory.go:55] Registering systemd factory
Jul 16 21:47:23.294714 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294716    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:47:23.294832 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.292136    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-124.ec2.internal.18c2e3692e7df0a1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-124.ec2.internal,UID:ip-10-0-143-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-124.ec2.internal,},FirstTimestamp:2026-07-16 21:47:23.283214497 +0000 UTC m=+0.330250076,LastTimestamp:2026-07-16 21:47:23.283214497 +0000 UTC m=+0.330250076,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-124.ec2.internal,}"
Jul 16 21:47:23.294975 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294961    2555 factory.go:153] Registering CRI-O factory
Jul 16 21:47:23.295006 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.294980    2555 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:47:23.295046 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.295036    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:47:23.295102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.295064    2555 factory.go:103] Registering Raw factory
Jul 16 21:47:23.295102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.295095    2555 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:47:23.295635 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.295536    2555 manager.go:349] Starting recovery of all containers
Jul 16 21:47:23.298556 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.298531    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9w58l"
Jul 16 21:47:23.304012 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.303965    2555 manager.go:354] Recovery completed
Jul 16 21:47:23.304779 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.304749    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:47:23.304779 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.304769    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:47:23.308506 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.308493    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.310278 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.310264    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.310317 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.310291    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.310317 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.310302    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.310812 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.310799    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:47:23.310868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.310812    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:47:23.310868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.310817    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.310868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.310830    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:47:23.311565 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.311547    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.311608 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.311575    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.311608 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.311586    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.312891 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.312878    2555 policy_none.go:49] "None policy: Start"
Jul 16 21:47:23.312953 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.312896    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:47:23.312953 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.312908    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:47:23.313584 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.313128    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-124.ec2.internal.18c2e369301ae8eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-124.ec2.internal,UID:ip-10-0-143-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-124.ec2.internal,},FirstTimestamp:2026-07-16 21:47:23.310278891 +0000 UTC m=+0.357314473,LastTimestamp:2026-07-16 21:47:23.310278891 +0000 UTC m=+0.357314473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-124.ec2.internal,}"
Jul 16 21:47:23.314792 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.314779    2555 policy_none.go:47] "Start"
Jul 16 21:47:23.317114 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.317096    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9w58l"
Jul 16 21:47:23.323831 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.323640    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-124.ec2.internal.18c2e369301b2c56  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-124.ec2.internal,UID:ip-10-0-143-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-124.ec2.internal,},FirstTimestamp:2026-07-16 21:47:23.31029615 +0000 UTC m=+0.357331728,LastTimestamp:2026-07-16 21:47:23.31029615 +0000 UTC m=+0.357331728,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-124.ec2.internal,}"
Jul 16 21:47:23.356499 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.356482    2555 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:47:23.356557 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.356527    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:47:23.356557 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.356540    2555 server.go:87] "Starting device plugin registration server"
Jul 16 21:47:23.356792 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.356780    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:47:23.356856 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.356796    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:47:23.356914 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.356896    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:47:23.357051 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.357035    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:47:23.357051 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.357047    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:47:23.357524 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.357499    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:47:23.357601 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.357551    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.433787 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.433749    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:47:23.436012 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.434972    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:47:23.436012 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.435007    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:47:23.436012 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.435034    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:47:23.436012 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.435076    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:47:23.437343 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.437321    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:23.457143 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.457116    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.457954 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.457939    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.458025 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.457969    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.458025 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.457979    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.458025 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.458002    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.467011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.466956    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.467011 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.466980    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-124.ec2.internal\": node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.485753 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.485733    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.535705 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.535675    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"]
Jul 16 21:47:23.535866 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.535764    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.536869 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.536846    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.536975 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.536879    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.536975 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.536893    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.539276 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.539263    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.539380 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.539365    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.539428 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.539397    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.540043 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.540029    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.540043 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.540037    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.540165 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.540056    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.540165 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.540057    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.540165 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.540066    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.540165 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.540070    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.542286 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.542268    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.542376 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.542296    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.543809 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.543792    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.543898 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.543820    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.543898 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.543831    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.565810 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.565787    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-124.ec2.internal\" not found" node="ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.570054 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.570037    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-124.ec2.internal\" not found" node="ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.585786 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.585766    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.595794 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.595775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dea81040f0d8cfba2ec3349c36ece347-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal\" (UID: \"dea81040f0d8cfba2ec3349c36ece347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.595896 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.595801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dea81040f0d8cfba2ec3349c36ece347-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal\" (UID: \"dea81040f0d8cfba2ec3349c36ece347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.595896 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.595819    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e21a0b953e6cd2a2d232f9eed463198-config\") pod \"kube-apiserver-proxy-ip-10-0-143-124.ec2.internal\" (UID: \"1e21a0b953e6cd2a2d232f9eed463198\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.686866 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.686831    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.696180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.696159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dea81040f0d8cfba2ec3349c36ece347-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal\" (UID: \"dea81040f0d8cfba2ec3349c36ece347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.696241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.696187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dea81040f0d8cfba2ec3349c36ece347-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal\" (UID: \"dea81040f0d8cfba2ec3349c36ece347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.696241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.696203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e21a0b953e6cd2a2d232f9eed463198-config\") pod \"kube-apiserver-proxy-ip-10-0-143-124.ec2.internal\" (UID: \"1e21a0b953e6cd2a2d232f9eed463198\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.696322 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.696241    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e21a0b953e6cd2a2d232f9eed463198-config\") pod \"kube-apiserver-proxy-ip-10-0-143-124.ec2.internal\" (UID: \"1e21a0b953e6cd2a2d232f9eed463198\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.696322 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.696267    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dea81040f0d8cfba2ec3349c36ece347-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal\" (UID: \"dea81040f0d8cfba2ec3349c36ece347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.696322 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.696270    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dea81040f0d8cfba2ec3349c36ece347-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal\" (UID: \"dea81040f0d8cfba2ec3349c36ece347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.787265 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.787185    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.868724 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.868698    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.873498 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:23.873437    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:23.888118 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.888101    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:23.988622 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:23.988591    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.089206 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.089127    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.189625 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.189590    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.234539 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.234516    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:47:24.234964 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.234665    2555 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.143.124:56766->44.215.225.81:6443: use of closed network connection"
Jul 16 21:47:24.234964 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.234677    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:47:24.234964 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.234733    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: unexpected error when reading response body. Please retry. Original error: read tcp 10.0.143.124:56766->44.215.225.81:6443: use of closed network connection" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:47:24.289667 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.289637    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.321543 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.321507    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:42:24 +0000 UTC" deadline="2028-03-20 21:20:27.623558454 +0000 UTC"
Jul 16 21:47:24.321683 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.321546    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14711h33m3.302027356s"
Jul 16 21:47:24.383313 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.383249    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:24.390358 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.390336    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.441442 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:24.441415    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea81040f0d8cfba2ec3349c36ece347.slice/crio-488c50121e79e290500bc7fb684f10e5adcd313d984ec73678433f1566ddcf5c WatchSource:0}: Error finding container 488c50121e79e290500bc7fb684f10e5adcd313d984ec73678433f1566ddcf5c: Status 404 returned error can't find the container with id 488c50121e79e290500bc7fb684f10e5adcd313d984ec73678433f1566ddcf5c
Jul 16 21:47:24.444643 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.444629    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:24.445711 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.445690    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:24.445785 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.445724    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:24.445785 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.445740    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:24.446385 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.446372    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:47:24.490807 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.490779    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.522963 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:24.522938    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:24.591160 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.591129    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.691736 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.691661    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.775964 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:24.775930    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21a0b953e6cd2a2d232f9eed463198.slice/crio-5c30eb994fa4834f997cb5dc4cb3b4673254613de512aee47ac2809495c72a39 WatchSource:0}: Error finding container 5c30eb994fa4834f997cb5dc4cb3b4673254613de512aee47ac2809495c72a39: Status 404 returned error can't find the container with id 5c30eb994fa4834f997cb5dc4cb3b4673254613de512aee47ac2809495c72a39
Jul 16 21:47:24.792274 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.792253    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.892733 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.892696    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:24.993214 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:24.993140    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.129890 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:26.129853    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:26.148052 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.148024    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.248431 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.248394    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.349133 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.349042    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.449628 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.449590    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.494630 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:26.494575    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal" event={"ID":"1e21a0b953e6cd2a2d232f9eed463198","Type":"ContainerStarted","Data":"5c30eb994fa4834f997cb5dc4cb3b4673254613de512aee47ac2809495c72a39"}
Jul 16 21:47:26.495778 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:26.495757    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal" event={"ID":"dea81040f0d8cfba2ec3349c36ece347","Type":"ContainerStarted","Data":"488c50121e79e290500bc7fb684f10e5adcd313d984ec73678433f1566ddcf5c"}
Jul 16 21:47:26.550100 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.550069    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.650659 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.650574    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.751635 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.751605    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.852451 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.852417    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:26.953096 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:26.953065    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.053628 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.053580    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.154170 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.154131    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.254714 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.254644    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.355465 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.355430    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.456194 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.456158    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.499913 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:27.499875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal" event={"ID":"dea81040f0d8cfba2ec3349c36ece347","Type":"ContainerStarted","Data":"b36a19612235018b31a0ca37c75383d5a997f56c0c407e8ad96d8e902156c512"}
Jul 16 21:47:27.500154 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:27.499969    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:27.501241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:27.501217    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:27.501342 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:27.501253    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:27.501342 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:27.501270    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:27.501471 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.501458    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-124.ec2.internal\" not found" node="ip-10-0-143-124.ec2.internal"
Jul 16 21:47:27.556890 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.556864    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.657339 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.657301    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.757924 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.757885    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.858713 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.858638    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:27.959320 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:27.959295    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-124.ec2.internal\" not found"
Jul 16 21:47:28.004399 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.004375    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:28.047665 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.047636    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:28.056648 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.056633    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:47:28.057611 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.057600    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:28.070714 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.070690    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:47:28.329866 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.329677    2555 apiserver.go:52] "Watching apiserver"
Jul 16 21:47:28.340045 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.340024    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:47:28.341060 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.341037    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-pzkkb","kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr","openshift-dns/node-resolver-rxtx7","openshift-image-registry/node-ca-8rmsf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal","openshift-multus/multus-additional-cni-plugins-bt8gz","openshift-cluster-node-tuning-operator/tuned-g8dtr","openshift-multus/multus-kp7tj","openshift-multus/network-metrics-daemon-7xsb4","openshift-network-diagnostics/network-check-target-6t5qm","openshift-network-operator/iptables-alerter-fn57s","openshift-ovn-kubernetes/ovnkube-node-97jzh","kube-system/global-pull-secret-syncer-c4l4q"]
Jul 16 21:47:28.343493 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.343478    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.345734 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.345711    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.347895 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.347871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.348361 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.348343    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:47:28.350172 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.350149    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.352293 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.352277    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.354597 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.354581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.354671 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.354634    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:47:28.354719 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.354654    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:28.354767 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.354637    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:47:28.354919 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.354901    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:47:28.355043 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.354923    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:47:28.356004 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.355944    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:47:28.356004 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.355980    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:47:28.356004 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.356000    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:47:28.356490 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.356475    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:47:28.356919 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.356904    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.357019 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.356972    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:47:28.357019 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.357008    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5dk5s\""
Jul 16 21:47:28.357263 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.357249    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:47:28.357331 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.357267    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jlqm2\""
Jul 16 21:47:28.357409 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.357266    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:47:28.357547 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.357533    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:47:28.358429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.358411    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:47:28.358516 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.358504    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:47:28.359306 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.359294    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.359513 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.359499    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cvhpb\""
Jul 16 21:47:28.360098 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.360082    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:47:28.360164 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.360103    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:47:28.361780 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.361763    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.363935 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.363919    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:28.364054 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.364014    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:28.366222 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.366194    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:28.366312 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.366243    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:28.368233 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.368214    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:47:28.369309 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.368900    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:47:28.369309 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369065    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:28.369309 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369151    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k5bvz\""
Jul 16 21:47:28.369309 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369285    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:47:28.369536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369463    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:47:28.369710 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369689    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:47:28.369776 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369752    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qj4t8\""
Jul 16 21:47:28.369878 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369861    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.369986 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369967    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:28.370050 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.369689    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mxf44\""
Jul 16 21:47:28.370313 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370294    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bzxw\""
Jul 16 21:47:28.370672 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovnkube-script-lib\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.370672 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2ms\" (UniqueName: \"kubernetes.io/projected/9770d49d-b509-486e-ac6e-8db8605082c6-kube-api-access-zk2ms\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.370672 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370587    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.370672 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370619    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-sys-fs\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.370672 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-tmp-dir\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.371134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.371134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovn-node-metrics-cert\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.371134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370816    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovnkube-config\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.371134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rp7\" (UniqueName: \"kubernetes.io/projected/2315cbbd-091f-45f8-9bd3-2d686e964eb1-kube-api-access-w4rp7\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.371134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.370870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.371685 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371664    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-system-cni-dir\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.371781 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371698    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-systemd\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.371781 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-cni-bin\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.371781 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371772    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-env-overrides\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.371944 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371810    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-registration-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.371944 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.371944 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-hosts-file\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.371944 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371897    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-slash\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.371944 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-var-lib-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-ovn\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.371985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7lp\" (UniqueName: \"kubernetes.io/projected/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-kube-api-access-dh7lp\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372016    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372039    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-systemd-units\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-run-netns\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-etc-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372113    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-cni-netd\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-socket-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-os-release\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372180    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.372201 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372202    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kq6q\" (UniqueName: \"kubernetes.io/projected/8e69ca4a-a391-454f-a7d1-129e509ef711-kube-api-access-8kq6q\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.372740 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-kubelet\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372740 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372263    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-node-log\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372740 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372293    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-device-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.372740 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-cnibin\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.372740 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372368    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-log-socket\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372740 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372400    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.372740 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.372431    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.376852 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.376837    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-df6cq\""
Jul 16 21:47:28.377435 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.377417    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:47:28.377709 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.377695    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:28.377765 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.377747    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:28.377846 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.377815    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:47:28.377846 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.377829    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:47:28.377989 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.377908    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47nfj\""
Jul 16 21:47:28.452238 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.449948    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:47:28.473345 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-cni-multus\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473350    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-multus-certs\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473370    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-cnibin\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473387    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473428    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473440    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-cnibin\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.473486 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2ms\" (UniqueName: \"kubernetes.io/projected/9770d49d-b509-486e-ac6e-8db8605082c6-kube-api-access-zk2ms\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473503    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d0e6cf0-0d52-4789-b236-cb1d36428c94-cni-binary-copy\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473529    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-netns\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-etc-kubernetes\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473578    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-cni-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473643    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-cni-bin\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473665    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlb5c\" (UniqueName: \"kubernetes.io/projected/9e81f22f-0b4d-41e2-82e9-0751cb36168c-kube-api-access-wlb5c\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473693    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/953e1a89-db86-4930-a0e5-ab7f21d009b7-iptables-alerter-script\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovnkube-config\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-system-cni-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.473771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-run\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473780    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-system-cni-dir\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-env-overrides\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-hosts-file\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473832    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-socket-dir-parent\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-system-cni-dir\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-hosts-file\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473898    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-kubelet\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2144dcd2-0967-4928-94ed-67201c9f687c-konnectivity-ca\") pod \"konnectivity-agent-pzkkb\" (UID: \"2144dcd2-0967-4928-94ed-67201c9f687c\") " pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysctl-conf\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.473993    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-var-lib-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474009    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7lp\" (UniqueName: \"kubernetes.io/projected/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-kube-api-access-dh7lp\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-var-lib-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-k8s-cni-cncf-io\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474062    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474079    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474095    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-systemd-units\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.474180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-etc-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-cni-netd\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474165    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-systemd-units\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474185    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-socket-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-cni-netd\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474210    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-sys-fs\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474234    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-cnibin\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474241    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-etc-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-os-release\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474319    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-sys-fs\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-socket-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kq6q\" (UniqueName: \"kubernetes.io/projected/8e69ca4a-a391-454f-a7d1-129e509ef711-kube-api-access-8kq6q\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474358    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-os-release\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-kubelet\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474392    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-kubelet\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-node-log\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474437    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-device-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-node-log\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-device-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474501    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2144dcd2-0967-4928-94ed-67201c9f687c-agent-certs\") pod \"konnectivity-agent-pzkkb\" (UID: \"2144dcd2-0967-4928-94ed-67201c9f687c\") " pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysconfig\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-log-socket\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovnkube-script-lib\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474596    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474631    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wds\" (UniqueName: \"kubernetes.io/projected/1d0e6cf0-0d52-4789-b236-cb1d36428c94-kube-api-access-v2wds\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73baff7b-e485-492f-97da-09da89b0cc69-host\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-log-socket\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474708    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-tmp-dir\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysctl-d\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-openvswitch\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474755    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e81f22f-0b4d-41e2-82e9-0751cb36168c-tmp\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/186438be-cc8d-4091-897c-f83f9acd1c5c-kubelet-config\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.475706 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/953e1a89-db86-4930-a0e5-ab7f21d009b7-host-slash\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73baff7b-e485-492f-97da-09da89b0cc69-serviceca\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474901    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756kw\" (UniqueName: \"kubernetes.io/projected/73baff7b-e485-492f-97da-09da89b0cc69-kube-api-access-756kw\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-env-overrides\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovn-node-metrics-cert\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.474987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-host\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475014    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475038    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rp7\" (UniqueName: \"kubernetes.io/projected/2315cbbd-091f-45f8-9bd3-2d686e964eb1-kube-api-access-w4rp7\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475061    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovnkube-config\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475062    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475073    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-tmp-dir\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475115    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-hostroot\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-conf-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-sys\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-systemd\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475192    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-cni-bin\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475254    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-registration-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovnkube-script-lib\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475267    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-systemd\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475273    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475290    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-kubernetes\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-registration-dir\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-cni-bin\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-var-lib-kubelet\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-tuned\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475411    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9770d49d-b509-486e-ac6e-8db8605082c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rmf\" (UniqueName: \"kubernetes.io/projected/1e18395e-950a-4347-a71c-e91f9bd7fc70-kube-api-access-56rmf\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-slash\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-ovn\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-os-release\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/186438be-cc8d-4091-897c-f83f9acd1c5c-dbus\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475510    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlfc\" (UniqueName: \"kubernetes.io/projected/953e1a89-db86-4930-a0e5-ab7f21d009b7-kube-api-access-zqlfc\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.476682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475508    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-slash\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-run-ovn\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475576    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-run-netns\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-daemon-config\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475619    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-modprobe-d\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-systemd\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-lib-modules\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475684    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2315cbbd-091f-45f8-9bd3-2d686e964eb1-host-run-netns\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e69ca4a-a391-454f-a7d1-129e509ef711-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.477144 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.475889    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e69ca4a-a391-454f-a7d1-129e509ef711-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.478502 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.478482    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2315cbbd-091f-45f8-9bd3-2d686e964eb1-ovn-node-metrics-cert\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.495718 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.495694    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7lp\" (UniqueName: \"kubernetes.io/projected/86f80ab0-9e2d-4881-b1c1-71fd4a113f8e-kube-api-access-dh7lp\") pod \"node-resolver-rxtx7\" (UID: \"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e\") " pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.495825 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.495793    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rp7\" (UniqueName: \"kubernetes.io/projected/2315cbbd-091f-45f8-9bd3-2d686e964eb1-kube-api-access-w4rp7\") pod \"ovnkube-node-97jzh\" (UID: \"2315cbbd-091f-45f8-9bd3-2d686e964eb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.495825 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.495817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2ms\" (UniqueName: \"kubernetes.io/projected/9770d49d-b509-486e-ac6e-8db8605082c6-kube-api-access-zk2ms\") pod \"aws-ebs-csi-driver-node-mvvhr\" (UID: \"9770d49d-b509-486e-ac6e-8db8605082c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.500331 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.500270    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kq6q\" (UniqueName: \"kubernetes.io/projected/8e69ca4a-a391-454f-a7d1-129e509ef711-kube-api-access-8kq6q\") pod \"multus-additional-cni-plugins-bt8gz\" (UID: \"8e69ca4a-a391-454f-a7d1-129e509ef711\") " pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.501331 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.501313    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-94fvd"
Jul 16 21:47:28.502673 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.502647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal" event={"ID":"1e21a0b953e6cd2a2d232f9eed463198","Type":"ContainerStarted","Data":"b79390bfe2e9d55b6e0ebae9e54755ebdbbf2d0f403b9b9769637bc6e687690d"}
Jul 16 21:47:28.502876 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.502863    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:28.502977 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.502948    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:28.514362 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.514348    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-94fvd"
Jul 16 21:47:28.514870 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.514855    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:47:28.514912 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.514880    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:47:28.514945 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.514922    2555 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-124.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:28.514945 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.514922    2555 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal"
Jul 16 21:47:28.576651 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2144dcd2-0967-4928-94ed-67201c9f687c-agent-certs\") pod \"konnectivity-agent-pzkkb\" (UID: \"2144dcd2-0967-4928-94ed-67201c9f687c\") " pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysconfig\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wds\" (UniqueName: \"kubernetes.io/projected/1d0e6cf0-0d52-4789-b236-cb1d36428c94-kube-api-access-v2wds\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576787    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73baff7b-e485-492f-97da-09da89b0cc69-host\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysctl-d\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e81f22f-0b4d-41e2-82e9-0751cb36168c-tmp\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/186438be-cc8d-4091-897c-f83f9acd1c5c-kubelet-config\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysconfig\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73baff7b-e485-492f-97da-09da89b0cc69-host\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.576868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576865    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/953e1a89-db86-4930-a0e5-ab7f21d009b7-host-slash\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73baff7b-e485-492f-97da-09da89b0cc69-serviceca\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576914    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/186438be-cc8d-4091-897c-f83f9acd1c5c-kubelet-config\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756kw\" (UniqueName: \"kubernetes.io/projected/73baff7b-e485-492f-97da-09da89b0cc69-kube-api-access-756kw\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/953e1a89-db86-4930-a0e5-ab7f21d009b7-host-slash\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.576991    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysctl-d\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577052    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-host\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577140    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577177    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-hostroot\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577200    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-conf-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-host\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-sys\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-kubernetes\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577335 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577340    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-sys\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-var-lib-kubelet\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577391    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-kubernetes\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577400    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-var-lib-kubelet\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-tuned\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-hostroot\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577443    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rmf\" (UniqueName: \"kubernetes.io/projected/1e18395e-950a-4347-a71c-e91f9bd7fc70-kube-api-access-56rmf\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-conf-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-os-release\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/186438be-cc8d-4091-897c-f83f9acd1c5c-dbus\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlfc\" (UniqueName: \"kubernetes.io/projected/953e1a89-db86-4930-a0e5-ab7f21d009b7-kube-api-access-zqlfc\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-daemon-config\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-modprobe-d\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-systemd\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577609    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-lib-modules\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-cni-multus\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-multus-certs\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.577931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-multus-certs\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577755    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-systemd\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d0e6cf0-0d52-4789-b236-cb1d36428c94-cni-binary-copy\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/186438be-cc8d-4091-897c-f83f9acd1c5c-dbus\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-netns\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-lib-modules\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577867    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-etc-kubernetes\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-etc-kubernetes\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577895    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73baff7b-e485-492f-97da-09da89b0cc69-serviceca\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577905    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-cni-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.577934    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577950    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-cni-multus\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-cni-bin\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.577979    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-cni-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlb5c\" (UniqueName: \"kubernetes.io/projected/9e81f22f-0b4d-41e2-82e9-0751cb36168c-kube-api-access-wlb5c\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578006    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-cni-bin\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.578026    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret podName:186438be-cc8d-4091-897c-f83f9acd1c5c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:29.078006261 +0000 UTC m=+5.071170408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret") pod "global-pull-secret-syncer-c4l4q" (UID: "186438be-cc8d-4091-897c-f83f9acd1c5c") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.578053    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:28.578754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-netns\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.578146    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs podName:1e18395e-950a-4347-a71c-e91f9bd7fc70 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:29.078133938 +0000 UTC m=+5.071298072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs") pod "network-metrics-daemon-7xsb4" (UID: "1e18395e-950a-4347-a71c-e91f9bd7fc70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/953e1a89-db86-4930-a0e5-ab7f21d009b7-iptables-alerter-script\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578194    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-system-cni-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-run\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578236    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-modprobe-d\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578273    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-run\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578145    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-os-release\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-system-cni-dir\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578339    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-socket-dir-parent\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578362    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-kubelet\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578372    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d0e6cf0-0d52-4789-b236-cb1d36428c94-cni-binary-copy\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2144dcd2-0967-4928-94ed-67201c9f687c-konnectivity-ca\") pod \"konnectivity-agent-pzkkb\" (UID: \"2144dcd2-0967-4928-94ed-67201c9f687c\") " pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysctl-conf\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578439    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-k8s-cni-cncf-io\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578446    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-socket-dir-parent\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:28.579358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-cnibin\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578537    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-var-lib-kubelet\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578580    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-cnibin\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d0e6cf0-0d52-4789-b236-cb1d36428c94-host-run-k8s-cni-cncf-io\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578722    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-sysctl-conf\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.578747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/953e1a89-db86-4930-a0e5-ab7f21d009b7-iptables-alerter-script\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.579190    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d0e6cf0-0d52-4789-b236-cb1d36428c94-multus-daemon-config\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.579648    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2144dcd2-0967-4928-94ed-67201c9f687c-agent-certs\") pod \"konnectivity-agent-pzkkb\" (UID: \"2144dcd2-0967-4928-94ed-67201c9f687c\") " pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.579690    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e81f22f-0b4d-41e2-82e9-0751cb36168c-etc-tuned\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.579765    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2144dcd2-0967-4928-94ed-67201c9f687c-konnectivity-ca\") pod \"konnectivity-agent-pzkkb\" (UID: \"2144dcd2-0967-4928-94ed-67201c9f687c\") " pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.579899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.579811    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e81f22f-0b4d-41e2-82e9-0751cb36168c-tmp\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.587928 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.587910    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:28.587989 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.587928    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:28.587989 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.587940    2555 projected.go:196] Error preparing data for projected volume kube-api-access-84m47 for pod openshift-network-diagnostics/network-check-target-6t5qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:28.588064 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:28.588034    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47 podName:46aaf5d5-4bb7-4fa0-ba40-47bc5329139b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:29.088019854 +0000 UTC m=+5.081183997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-84m47" (UniqueName: "kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47") pod "network-check-target-6t5qm" (UID: "46aaf5d5-4bb7-4fa0-ba40-47bc5329139b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:28.591508 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.591486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rmf\" (UniqueName: \"kubernetes.io/projected/1e18395e-950a-4347-a71c-e91f9bd7fc70-kube-api-access-56rmf\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:28.592086 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.592058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlb5c\" (UniqueName: \"kubernetes.io/projected/9e81f22f-0b4d-41e2-82e9-0751cb36168c-kube-api-access-wlb5c\") pod \"tuned-g8dtr\" (UID: \"9e81f22f-0b4d-41e2-82e9-0751cb36168c\") " pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.592915 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.592890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlfc\" (UniqueName: \"kubernetes.io/projected/953e1a89-db86-4930-a0e5-ab7f21d009b7-kube-api-access-zqlfc\") pod \"iptables-alerter-fn57s\" (UID: \"953e1a89-db86-4930-a0e5-ab7f21d009b7\") " pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.593021 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.593004    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-756kw\" (UniqueName: \"kubernetes.io/projected/73baff7b-e485-492f-97da-09da89b0cc69-kube-api-access-756kw\") pod \"node-ca-8rmsf\" (UID: \"73baff7b-e485-492f-97da-09da89b0cc69\") " pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.593365 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.593343    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wds\" (UniqueName: \"kubernetes.io/projected/1d0e6cf0-0d52-4789-b236-cb1d36428c94-kube-api-access-v2wds\") pod \"multus-kp7tj\" (UID: \"1d0e6cf0-0d52-4789-b236-cb1d36428c94\") " pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.653326 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.653292    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bt8gz"
Jul 16 21:47:28.657996 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.657981    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:28.662919 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.662894    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e69ca4a_a391_454f_a7d1_129e509ef711.slice/crio-1893b5bce76e625c63332b7a16df768a3b67a2ae5039c49619eeadaecf9cb0f2 WatchSource:0}: Error finding container 1893b5bce76e625c63332b7a16df768a3b67a2ae5039c49619eeadaecf9cb0f2: Status 404 returned error can't find the container with id 1893b5bce76e625c63332b7a16df768a3b67a2ae5039c49619eeadaecf9cb0f2
Jul 16 21:47:28.664758 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.664738    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr"
Jul 16 21:47:28.666244 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.666226    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2315cbbd_091f_45f8_9bd3_2d686e964eb1.slice/crio-fe5ae774be05e1845371641caf1386b76fd6dc7b7b434cc3bea8196431ea2faf WatchSource:0}: Error finding container fe5ae774be05e1845371641caf1386b76fd6dc7b7b434cc3bea8196431ea2faf: Status 404 returned error can't find the container with id fe5ae774be05e1845371641caf1386b76fd6dc7b7b434cc3bea8196431ea2faf
Jul 16 21:47:28.672772 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.672748    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rxtx7"
Jul 16 21:47:28.673565 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.673545    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9770d49d_b509_486e_ac6e_8db8605082c6.slice/crio-1a28b4d22189ec811054e510b76b9d0f65fecca185298c89fda7c69a796e2716 WatchSource:0}: Error finding container 1a28b4d22189ec811054e510b76b9d0f65fecca185298c89fda7c69a796e2716: Status 404 returned error can't find the container with id 1a28b4d22189ec811054e510b76b9d0f65fecca185298c89fda7c69a796e2716
Jul 16 21:47:28.678501 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.678484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8rmsf"
Jul 16 21:47:28.681573 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.681552    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f80ab0_9e2d_4881_b1c1_71fd4a113f8e.slice/crio-61f32a89f8db2eddd4f7c76af4483774f9d890be6d05aadaeb404b860748c096 WatchSource:0}: Error finding container 61f32a89f8db2eddd4f7c76af4483774f9d890be6d05aadaeb404b860748c096: Status 404 returned error can't find the container with id 61f32a89f8db2eddd4f7c76af4483774f9d890be6d05aadaeb404b860748c096
Jul 16 21:47:28.683938 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.683915    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kp7tj"
Jul 16 21:47:28.686808 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.686789    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73baff7b_e485_492f_97da_09da89b0cc69.slice/crio-fbb8969c68167ebe41d9976d033b92422ad4122ce073299625c2a9f6a65ad013 WatchSource:0}: Error finding container fbb8969c68167ebe41d9976d033b92422ad4122ce073299625c2a9f6a65ad013: Status 404 returned error can't find the container with id fbb8969c68167ebe41d9976d033b92422ad4122ce073299625c2a9f6a65ad013
Jul 16 21:47:28.688705 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.688688    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:28.693684 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.693667    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8dtr"
Jul 16 21:47:28.694028 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.694010    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e6cf0_0d52_4789_b236_cb1d36428c94.slice/crio-d333fc8967c563e9e280f3e58ac56382eb728f61fe930159320033c8202eefba WatchSource:0}: Error finding container d333fc8967c563e9e280f3e58ac56382eb728f61fe930159320033c8202eefba: Status 404 returned error can't find the container with id d333fc8967c563e9e280f3e58ac56382eb728f61fe930159320033c8202eefba
Jul 16 21:47:28.699660 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.699022    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fn57s"
Jul 16 21:47:28.705591 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.705570    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e81f22f_0b4d_41e2_82e9_0751cb36168c.slice/crio-10d155fb75db007cf64ae7b88c37563622726b842501d7edbdf0158c6cd0c796 WatchSource:0}: Error finding container 10d155fb75db007cf64ae7b88c37563622726b842501d7edbdf0158c6cd0c796: Status 404 returned error can't find the container with id 10d155fb75db007cf64ae7b88c37563622726b842501d7edbdf0158c6cd0c796
Jul 16 21:47:28.708395 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:47:28.708376    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953e1a89_db86_4930_a0e5_ab7f21d009b7.slice/crio-fd8ab6be835975eb2b1476dc36975dc0f3b87773e0c7ccde82ffea6c5d56b83d WatchSource:0}: Error finding container fd8ab6be835975eb2b1476dc36975dc0f3b87773e0c7ccde82ffea6c5d56b83d: Status 404 returned error can't find the container with id fd8ab6be835975eb2b1476dc36975dc0f3b87773e0c7ccde82ffea6c5d56b83d
Jul 16 21:47:28.711829 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:28.711784    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-124.ec2.internal" podStartSLOduration=0.711772875 podStartE2EDuration="711.772875ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:28.711672315 +0000 UTC m=+4.704836469" watchObservedRunningTime="2026-07-16 21:47:28.711772875 +0000 UTC m=+4.704937027"
Jul 16 21:47:29.083066 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.083023    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:29.083066 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.083084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:29.083302 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.083183    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:29.083302 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.083215    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:29.083302 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.083254    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret podName:186438be-cc8d-4091-897c-f83f9acd1c5c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:30.083234224 +0000 UTC m=+6.076398354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret") pod "global-pull-secret-syncer-c4l4q" (UID: "186438be-cc8d-4091-897c-f83f9acd1c5c") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:29.083302 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.083275    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs podName:1e18395e-950a-4347-a71c-e91f9bd7fc70 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:30.083265329 +0000 UTC m=+6.076429462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs") pod "network-metrics-daemon-7xsb4" (UID: "1e18395e-950a-4347-a71c-e91f9bd7fc70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:29.183799 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.183760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:29.183993 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.183909    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:29.183993 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.183928    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:29.183993 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.183939    2555 projected.go:196] Error preparing data for projected volume kube-api-access-84m47 for pod openshift-network-diagnostics/network-check-target-6t5qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:29.184160 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.184008    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47 podName:46aaf5d5-4bb7-4fa0-ba40-47bc5329139b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:30.183988946 +0000 UTC m=+6.177153090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-84m47" (UniqueName: "kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47") pod "network-check-target-6t5qm" (UID: "46aaf5d5-4bb7-4fa0-ba40-47bc5329139b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:29.489693 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.489665    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:29.490152 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:29.489790    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:29.514014 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.513979    2555 generic.go:356] "Generic (PLEG): container finished" podID="dea81040f0d8cfba2ec3349c36ece347" containerID="b36a19612235018b31a0ca37c75383d5a997f56c0c407e8ad96d8e902156c512" exitCode=0
Jul 16 21:47:29.514173 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.514091    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal" event={"ID":"dea81040f0d8cfba2ec3349c36ece347","Type":"ContainerDied","Data":"b36a19612235018b31a0ca37c75383d5a997f56c0c407e8ad96d8e902156c512"}
Jul 16 21:47:29.519013 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.518902    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:42:28 +0000 UTC" deadline="2028-04-21 01:50:30.951107294 +0000 UTC"
Jul 16 21:47:29.519013 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.518948    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15460h3m1.432162527s"
Jul 16 21:47:29.520724 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.520689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pzkkb" event={"ID":"2144dcd2-0967-4928-94ed-67201c9f687c","Type":"ContainerStarted","Data":"765b174c7a9c3dabd94b92bcf71b5376ffdbd71d42be6e295e7eab6ecfa4b38e"}
Jul 16 21:47:29.532234 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.532207    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8rmsf" event={"ID":"73baff7b-e485-492f-97da-09da89b0cc69","Type":"ContainerStarted","Data":"fbb8969c68167ebe41d9976d033b92422ad4122ce073299625c2a9f6a65ad013"}
Jul 16 21:47:29.545561 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.545529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rxtx7" event={"ID":"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e","Type":"ContainerStarted","Data":"61f32a89f8db2eddd4f7c76af4483774f9d890be6d05aadaeb404b860748c096"}
Jul 16 21:47:29.557151 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.557097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr" event={"ID":"9770d49d-b509-486e-ac6e-8db8605082c6","Type":"ContainerStarted","Data":"1a28b4d22189ec811054e510b76b9d0f65fecca185298c89fda7c69a796e2716"}
Jul 16 21:47:29.574013 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.573984    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fn57s" event={"ID":"953e1a89-db86-4930-a0e5-ab7f21d009b7","Type":"ContainerStarted","Data":"fd8ab6be835975eb2b1476dc36975dc0f3b87773e0c7ccde82ffea6c5d56b83d"}
Jul 16 21:47:29.585048 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.585014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8dtr" event={"ID":"9e81f22f-0b4d-41e2-82e9-0751cb36168c","Type":"ContainerStarted","Data":"10d155fb75db007cf64ae7b88c37563622726b842501d7edbdf0158c6cd0c796"}
Jul 16 21:47:29.590058 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.590032    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kp7tj" event={"ID":"1d0e6cf0-0d52-4789-b236-cb1d36428c94","Type":"ContainerStarted","Data":"d333fc8967c563e9e280f3e58ac56382eb728f61fe930159320033c8202eefba"}
Jul 16 21:47:29.603342 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.603305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"fe5ae774be05e1845371641caf1386b76fd6dc7b7b434cc3bea8196431ea2faf"}
Jul 16 21:47:29.609294 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:29.609243    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerStarted","Data":"1893b5bce76e625c63332b7a16df768a3b67a2ae5039c49619eeadaecf9cb0f2"}
Jul 16 21:47:30.091334 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.091248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:30.091334 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.091310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:30.091568 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.091442    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:30.091568 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.091502    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs podName:1e18395e-950a-4347-a71c-e91f9bd7fc70 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:32.0914858 +0000 UTC m=+8.084649950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs") pod "network-metrics-daemon-7xsb4" (UID: "1e18395e-950a-4347-a71c-e91f9bd7fc70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:30.091568 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.091566    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:30.091755 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.091600    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret podName:186438be-cc8d-4091-897c-f83f9acd1c5c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:32.09158801 +0000 UTC m=+8.084752263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret") pod "global-pull-secret-syncer-c4l4q" (UID: "186438be-cc8d-4091-897c-f83f9acd1c5c") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:30.192061 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.192016    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:30.192250 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.192172    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:30.192250 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.192200    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:30.192250 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.192214    2555 projected.go:196] Error preparing data for projected volume kube-api-access-84m47 for pod openshift-network-diagnostics/network-check-target-6t5qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:30.192410 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.192282    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47 podName:46aaf5d5-4bb7-4fa0-ba40-47bc5329139b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:32.1922614 +0000 UTC m=+8.185425543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-84m47" (UniqueName: "kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47") pod "network-check-target-6t5qm" (UID: "46aaf5d5-4bb7-4fa0-ba40-47bc5329139b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:30.490702 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.490092    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:30.490702 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.490231    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:30.491523 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.491333    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:30.491523 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:30.491438    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:30.519641 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.519599    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:42:28 +0000 UTC" deadline="2028-03-19 03:22:58.794192589 +0000 UTC"
Jul 16 21:47:30.519641 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.519639    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14669h35m28.274558788s"
Jul 16 21:47:30.628987 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.628323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal" event={"ID":"dea81040f0d8cfba2ec3349c36ece347","Type":"ContainerStarted","Data":"8e74f46ac55807d47bb68afbf5ab43b3ec0f096c4a6845d5c416561837b7ffcc"}
Jul 16 21:47:30.647017 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:30.646218    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-124.ec2.internal" podStartSLOduration=2.646196907 podStartE2EDuration="2.646196907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:30.645937135 +0000 UTC m=+6.639101289" watchObservedRunningTime="2026-07-16 21:47:30.646196907 +0000 UTC m=+6.639361060"
Jul 16 21:47:31.490487 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:31.489976    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:31.490487 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:31.490114    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:32.109755 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:32.109019    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:32.109755 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:32.109090    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:32.109755 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.109227    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:32.109755 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.109292    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs podName:1e18395e-950a-4347-a71c-e91f9bd7fc70 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:36.109271456 +0000 UTC m=+12.102435604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs") pod "network-metrics-daemon-7xsb4" (UID: "1e18395e-950a-4347-a71c-e91f9bd7fc70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:32.109755 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.109368    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:32.109755 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.109407    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret podName:186438be-cc8d-4091-897c-f83f9acd1c5c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:36.109395493 +0000 UTC m=+12.102559623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret") pod "global-pull-secret-syncer-c4l4q" (UID: "186438be-cc8d-4091-897c-f83f9acd1c5c") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:32.209623 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:32.209575    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:32.210006 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.209782    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:32.210006 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.209805    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:32.210006 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.209817    2555 projected.go:196] Error preparing data for projected volume kube-api-access-84m47 for pod openshift-network-diagnostics/network-check-target-6t5qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:32.210006 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.209873    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47 podName:46aaf5d5-4bb7-4fa0-ba40-47bc5329139b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:36.209856599 +0000 UTC m=+12.203020729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-84m47" (UniqueName: "kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47") pod "network-check-target-6t5qm" (UID: "46aaf5d5-4bb7-4fa0-ba40-47bc5329139b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:32.492987 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:32.492911    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:32.493243 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.493061    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:32.493243 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:32.493130    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:32.493356 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:32.493253    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:33.489505 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:33.489473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:33.490088 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:33.489615    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:34.490919 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:34.490308    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:34.490919 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:34.490434    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:34.490919 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:34.490790    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:34.490919 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:34.490874    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:35.489729 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:35.489693    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:35.489917 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:35.489832    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:36.143179 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:36.143145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:36.143658 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:36.143195    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:36.143658 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.143312    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:36.143658 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.143392    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret podName:186438be-cc8d-4091-897c-f83f9acd1c5c nodeName:}" failed. No retries permitted until 2026-07-16 21:47:44.143370963 +0000 UTC m=+20.136535093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret") pod "global-pull-secret-syncer-c4l4q" (UID: "186438be-cc8d-4091-897c-f83f9acd1c5c") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:36.143658 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.143314    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:36.143658 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.143458    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs podName:1e18395e-950a-4347-a71c-e91f9bd7fc70 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:44.143439196 +0000 UTC m=+20.136603343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs") pod "network-metrics-daemon-7xsb4" (UID: "1e18395e-950a-4347-a71c-e91f9bd7fc70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:36.244459 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:36.244420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:36.244641 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.244587    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:36.244641 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.244611    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:36.244641 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.244625    2555 projected.go:196] Error preparing data for projected volume kube-api-access-84m47 for pod openshift-network-diagnostics/network-check-target-6t5qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:36.244795 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.244694    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47 podName:46aaf5d5-4bb7-4fa0-ba40-47bc5329139b nodeName:}" failed. No retries permitted until 2026-07-16 21:47:44.244674969 +0000 UTC m=+20.237839104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-84m47" (UniqueName: "kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47") pod "network-check-target-6t5qm" (UID: "46aaf5d5-4bb7-4fa0-ba40-47bc5329139b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:36.490161 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:36.490074    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:36.490330 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.490226    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:36.490330 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:36.490316    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:36.490474 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:36.490418    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:37.490461 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:37.490114    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:37.490461 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:37.490249    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:38.489530 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:38.489497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:38.489696 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:38.489546    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:38.489696 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:38.489646    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:38.493372 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:38.489777    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:39.489490 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:39.489449    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:39.489675 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:39.489564    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:40.490193 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:40.490155    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:40.490652 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:40.490294    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:40.490652 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:40.490351    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:40.490652 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:40.490456    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:41.489299 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:41.489258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:41.489460 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:41.489374    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:42.489552 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:42.489512    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:42.490023 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:42.489674    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:42.490023 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:42.489691    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:42.490023 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:42.489829    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:43.489290 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:43.489245    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:43.489481 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:43.489385    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:44.203199 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:44.203162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:44.203641 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:44.203208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:44.203641 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.203347    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:44.203641 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.203418    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:44.203641 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.203435    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret podName:186438be-cc8d-4091-897c-f83f9acd1c5c nodeName:}" failed. No retries permitted until 2026-07-16 21:48:00.203411309 +0000 UTC m=+36.196575444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret") pod "global-pull-secret-syncer-c4l4q" (UID: "186438be-cc8d-4091-897c-f83f9acd1c5c") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:44.203641 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.203480    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs podName:1e18395e-950a-4347-a71c-e91f9bd7fc70 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:00.203462192 +0000 UTC m=+36.196626324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs") pod "network-metrics-daemon-7xsb4" (UID: "1e18395e-950a-4347-a71c-e91f9bd7fc70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:44.304572 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:44.304533    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:44.304774 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.304684    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:44.304774 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.304709    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:44.304774 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.304722    2555 projected.go:196] Error preparing data for projected volume kube-api-access-84m47 for pod openshift-network-diagnostics/network-check-target-6t5qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:44.304931 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.304785    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47 podName:46aaf5d5-4bb7-4fa0-ba40-47bc5329139b nodeName:}" failed. No retries permitted until 2026-07-16 21:48:00.304768808 +0000 UTC m=+36.297932956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-84m47" (UniqueName: "kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47") pod "network-check-target-6t5qm" (UID: "46aaf5d5-4bb7-4fa0-ba40-47bc5329139b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:44.491032 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:44.490937    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:44.491032 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:44.490992    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:44.491238 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.491076    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:44.491238 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:44.491208    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:45.489274 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:45.489242    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:45.489693 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:45.489342    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:46.489571 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:46.489536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:46.490187 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:46.489536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:46.490187 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:46.489683    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:46.490187 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:46.489765    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:47.489461 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:47.489430    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:47.489650 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:47.489535    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:47.665031 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:47.663838    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pzkkb" event={"ID":"2144dcd2-0967-4928-94ed-67201c9f687c","Type":"ContainerStarted","Data":"462a69ad8bb4095b9a7766f576cf7dbd61bd3098a20c7871b1a4d3c03e6a6c5b"}
Jul 16 21:47:48.490198 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.489995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:48.490904 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.489995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:48.490904 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:48.490255    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:48.490904 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:48.490336    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:48.668084 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.667968    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e69ca4a-a391-454f-a7d1-129e509ef711" containerID="10f4d31bd245ff40ccb7db1b7cbd33298f4a7762c7069f96582ade6b5d7342bc" exitCode=0
Jul 16 21:47:48.668084 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.668001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerDied","Data":"10f4d31bd245ff40ccb7db1b7cbd33298f4a7762c7069f96582ade6b5d7342bc"}
Jul 16 21:47:48.669904 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.669874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8rmsf" event={"ID":"73baff7b-e485-492f-97da-09da89b0cc69","Type":"ContainerStarted","Data":"cf284b55e12df33a930505aee90a0d3c1292737947127a6e21485a4e31d6938a"}
Jul 16 21:47:48.671541 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.671514    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rxtx7" event={"ID":"86f80ab0-9e2d-4881-b1c1-71fd4a113f8e","Type":"ContainerStarted","Data":"8457171e10a85a6584207166cdc2f5a02e3232fa82d7c932e490ca9ee6658f24"}
Jul 16 21:47:48.673010 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.672987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr" event={"ID":"9770d49d-b509-486e-ac6e-8db8605082c6","Type":"ContainerStarted","Data":"beebe70c6e9de922d7c811c017debda83e906505114df19967e2e430cc6d5ca0"}
Jul 16 21:47:48.674934 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.674912    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8dtr" event={"ID":"9e81f22f-0b4d-41e2-82e9-0751cb36168c","Type":"ContainerStarted","Data":"38e8d0db980a72818e64fca6105f3f71fa50372d7aecdcce81ce84c753209f65"}
Jul 16 21:47:48.676766 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.676738    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kp7tj" event={"ID":"1d0e6cf0-0d52-4789-b236-cb1d36428c94","Type":"ContainerStarted","Data":"384d81b13e6915ab946a174458e69cbf7c366075f8ac44c60367db1298ed9895"}
Jul 16 21:47:48.680747 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.680727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"d3b3a91e35635c161b9aeba387247fb387bf43bb65f7f5e452e62a5e642faa39"}
Jul 16 21:47:48.680910 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.680894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"0d2cf32c3ae2a1f7c1bd39e5e91b9b7f38cc36880a1a99ff77790300b8514fc3"}
Jul 16 21:47:48.681068 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.681048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"b204393cb3ed0ad53822322e4a936894c7db07a6a8d9559c60c58a0addcf2d8e"}
Jul 16 21:47:48.681172 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.681098    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"d4c03bbddbc32b52892e3ed7c5af3360c6052fe563cf7815cb64c318d3da91fe"}
Jul 16 21:47:48.681172 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.681138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"d885a79dffd53ac5f69ce6f38fbdc606922223202f9f13448d10d418e5d7259b"}
Jul 16 21:47:48.681285 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.681177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"b609d147356879956127a4917890da16048faaec68c3daa3ceb54acd7a90f344"}
Jul 16 21:47:48.693693 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.693656    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pzkkb" podStartSLOduration=14.433903355 podStartE2EDuration="24.693642484s" totalImagesPullingTime="10.259739129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:47.685026965 +0000 UTC m=+23.678191119" watchObservedRunningTime="2026-07-16 21:47:48.693642484 +0000 UTC m=+24.686806635"
Jul 16 21:47:48.712128 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.712079    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kp7tj" podStartSLOduration=5.870108429 podStartE2EDuration="24.71206607s" totalImagesPullingTime="18.841957641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:48.711547981 +0000 UTC m=+24.704712127" watchObservedRunningTime="2026-07-16 21:47:48.71206607 +0000 UTC m=+24.705230222"
Jul 16 21:47:48.734575 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.734530    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g8dtr" podStartSLOduration=5.91980098 podStartE2EDuration="24.734511731s" totalImagesPullingTime="18.814710751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:48.734229228 +0000 UTC m=+24.727393380" watchObservedRunningTime="2026-07-16 21:47:48.734511731 +0000 UTC m=+24.727675884"
Jul 16 21:47:48.738078 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.738061    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:47:48.751975 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:48.751887    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rxtx7" podStartSLOduration=5.915619982 podStartE2EDuration="24.751874206s" totalImagesPullingTime="18.836254224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:48.751403566 +0000 UTC m=+24.744567717" watchObservedRunningTime="2026-07-16 21:47:48.751874206 +0000 UTC m=+24.745038357"
Jul 16 21:47:49.421899 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.421795    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:47:48.738075623Z","UUID":"47bdbcbb-78ac-4544-b7b6-36733536c69a","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:47:49.423912 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.423877    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:47:49.423912 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.423916    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:47:49.489631 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.489602    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:49.489801 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:49.489725    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:49.685237 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.685063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fn57s" event={"ID":"953e1a89-db86-4930-a0e5-ab7f21d009b7","Type":"ContainerStarted","Data":"443a79dbcf65d5be18b9d429c8377b3a3fb2adce712a8b6af74d13eb717f4eb0"}
Jul 16 21:47:49.687435 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.687351    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr" event={"ID":"9770d49d-b509-486e-ac6e-8db8605082c6","Type":"ContainerStarted","Data":"9be89e843d91a99d08b733fc853465211762c2ef4e9f78405654e472fc2211f9"}
Jul 16 21:47:49.700782 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.700733    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8rmsf" podStartSLOduration=6.870881886 podStartE2EDuration="25.700712799s" totalImagesPullingTime="18.829830913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:48.767380941 +0000 UTC m=+24.760545104" watchObservedRunningTime="2026-07-16 21:47:49.700712799 +0000 UTC m=+25.693876952"
Jul 16 21:47:49.701312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:49.701276    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fn57s" podStartSLOduration=6.890888401 podStartE2EDuration="25.701262663s" totalImagesPullingTime="18.810374262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:49.70040403 +0000 UTC m=+25.693568183" watchObservedRunningTime="2026-07-16 21:47:49.701262663 +0000 UTC m=+25.694426816"
Jul 16 21:47:50.430329 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.430069    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:50.430758 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.430735    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:50.490132 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.490103    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:50.490312 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:50.490245    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:50.490312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.490277    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:50.490428 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:50.490405    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:50.694109 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.694076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"dc1cd936caeafd027bf970710feff919042c33a00346a12c716ae5e920d128be"}
Jul 16 21:47:50.696453 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.696421    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr" event={"ID":"9770d49d-b509-486e-ac6e-8db8605082c6","Type":"ContainerStarted","Data":"a345121736bcf85bd2011c37676fb4ab4ae46b00cf21919f0b990ed22a8d2cb6"}
Jul 16 21:47:50.696819 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.696801    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:50.697296 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.697198    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pzkkb"
Jul 16 21:47:50.715301 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:50.715234    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvvhr" podStartSLOduration=6.073417283 podStartE2EDuration="26.715218506s" totalImagesPullingTime="20.641801223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:50.715177453 +0000 UTC m=+26.708341602" watchObservedRunningTime="2026-07-16 21:47:50.715218506 +0000 UTC m=+26.708382658"
Jul 16 21:47:51.489525 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:51.489496    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:51.489744 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:51.489614    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:52.489530 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.489493    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:52.490214 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.489493    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:52.490214 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:52.489654    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:52.490214 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:52.489717    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:52.704982 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.704706    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" event={"ID":"2315cbbd-091f-45f8-9bd3-2d686e964eb1","Type":"ContainerStarted","Data":"69ee1b31c120f07550e3fbfb72a775d754f5ebd865ae6f1bc079ab480eefe723"}
Jul 16 21:47:52.705130 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.705077    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:52.705178 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.705132    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:52.705178 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.705175    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:52.707213 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.707174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerStarted","Data":"a3f7b726a585e12dc426bc00d7f847cfff78bed2ac3a9f5534bd1aa3ee079759"}
Jul 16 21:47:52.721813 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.721777    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:52.724053 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.724031    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:47:52.739917 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:52.739868    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh" podStartSLOduration=9.510714047 podStartE2EDuration="28.739854019s" totalImagesPullingTime="19.229139972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:52.739325061 +0000 UTC m=+28.732489212" watchObservedRunningTime="2026-07-16 21:47:52.739854019 +0000 UTC m=+28.733018171"
Jul 16 21:47:53.489633 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:53.489599    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:53.490218 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:53.489703    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:53.710675 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:53.710642    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e69ca4a-a391-454f-a7d1-129e509ef711" containerID="a3f7b726a585e12dc426bc00d7f847cfff78bed2ac3a9f5534bd1aa3ee079759" exitCode=0
Jul 16 21:47:53.710834 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:53.710718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerDied","Data":"a3f7b726a585e12dc426bc00d7f847cfff78bed2ac3a9f5534bd1aa3ee079759"}
Jul 16 21:47:54.490618 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.490589    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:54.491093 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.490650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:54.491093 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:54.490721    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:54.491093 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:54.490786    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:54.539541 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.539459    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6t5qm"]
Jul 16 21:47:54.543190 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.543164    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7xsb4"]
Jul 16 21:47:54.549180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.549109    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c4l4q"]
Jul 16 21:47:54.549294 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.549233    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:54.549370 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:54.549344    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:54.715593 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.715411    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e69ca4a-a391-454f-a7d1-129e509ef711" containerID="1c099f721db292df28266814267a9c95ed742f83e45074f25535529c682ff9c5" exitCode=0
Jul 16 21:47:54.715731 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.715643    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:54.715731 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.715496    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerDied","Data":"1c099f721db292df28266814267a9c95ed742f83e45074f25535529c682ff9c5"}
Jul 16 21:47:54.715826 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:54.715752    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:54.715898 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:54.715882    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:54.716464 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:54.716293    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:55.720125 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:55.720044    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e69ca4a-a391-454f-a7d1-129e509ef711" containerID="0843882fc1da9de658b7d00e14f9407d7810491ad8d549743c2254f3723e8127" exitCode=0
Jul 16 21:47:55.720125 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:55.720094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerDied","Data":"0843882fc1da9de658b7d00e14f9407d7810491ad8d549743c2254f3723e8127"}
Jul 16 21:47:56.489478 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:56.489439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:56.489478 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:56.489462    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:56.489636 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:56.489439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:56.489636 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:56.489600    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:47:56.489727 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:56.489668    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:56.489771 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:56.489756    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:58.490079 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:58.490020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:47:58.490079 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:58.490059    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:47:58.490904 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:47:58.490020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:47:58.490904 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:58.490149    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c4l4q" podUID="186438be-cc8d-4091-897c-f83f9acd1c5c"
Jul 16 21:47:58.490904 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:58.490223    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6t5qm" podUID="46aaf5d5-4bb7-4fa0-ba40-47bc5329139b"
Jul 16 21:47:58.490904 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:47:58.490300    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xsb4" podUID="1e18395e-950a-4347-a71c-e91f9bd7fc70"
Jul 16 21:48:00.224054 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.224034    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:48:00.224465 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.224071    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:48:00.224562 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.224547    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:48:00.224611 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.224562    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:48:00.224611 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.224610    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs podName:1e18395e-950a-4347-a71c-e91f9bd7fc70 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:32.224592378 +0000 UTC m=+68.217756528 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs") pod "network-metrics-daemon-7xsb4" (UID: "1e18395e-950a-4347-a71c-e91f9bd7fc70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:48:00.224682 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.224624    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret podName:186438be-cc8d-4091-897c-f83f9acd1c5c nodeName:}" failed. No retries permitted until 2026-07-16 21:48:32.224618339 +0000 UTC m=+68.217782469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret") pod "global-pull-secret-syncer-c4l4q" (UID: "186438be-cc8d-4091-897c-f83f9acd1c5c") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:48:00.313431 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.313411    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-124.ec2.internal" event="NodeReady"
Jul 16 21:48:00.313586 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.313573    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:48:00.324756 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.324725    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:48:00.324894 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.324876    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:48:00.324934 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.324900    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:48:00.324934 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.324909    2555 projected.go:196] Error preparing data for projected volume kube-api-access-84m47 for pod openshift-network-diagnostics/network-check-target-6t5qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:48:00.325035 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.324972    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47 podName:46aaf5d5-4bb7-4fa0-ba40-47bc5329139b nodeName:}" failed. No retries permitted until 2026-07-16 21:48:32.324942796 +0000 UTC m=+68.318106926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-84m47" (UniqueName: "kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47") pod "network-check-target-6t5qm" (UID: "46aaf5d5-4bb7-4fa0-ba40-47bc5329139b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:48:00.393384 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.393356    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xshcr"]
Jul 16 21:48:00.414283 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.414254    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-767c4f9cc7-gqnqw"]
Jul 16 21:48:00.414407 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.414387    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.419171 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.419004    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:48:00.419171 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.419036    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:48:00.420130 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.420110    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9tv9m\""
Jul 16 21:48:00.433305 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.433288    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v76p8"]
Jul 16 21:48:00.433415 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.433398    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.436477 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.436462    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:48:00.436576 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.436502    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:48:00.436576 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.436552    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:48:00.436653 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.436561    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdfjl\""
Jul 16 21:48:00.443872 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.443856    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:48:00.449454 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.449434    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xshcr"]
Jul 16 21:48:00.449551 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.449463    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v76p8"]
Jul 16 21:48:00.449618 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.449565    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:00.454343 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.454322    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fbkjh\""
Jul 16 21:48:00.454571 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.454553    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:48:00.454571 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.454566    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:48:00.454971 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.454942    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-767c4f9cc7-gqnqw"]
Jul 16 21:48:00.458449 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.458432    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:48:00.489485 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.489464    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:48:00.489615 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.489600    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:48:00.489732 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.489466    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:48:00.492100 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.492079    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mc6l9\""
Jul 16 21:48:00.492886 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.492868    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:48:00.493030 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.493014    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:48:00.493346 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.493331    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:48:00.493491 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.493477    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6g595\""
Jul 16 21:48:00.493598 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.493585    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:48:00.526313 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526288    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-config-volume\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.526410 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526321    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.526410 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9ff\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-kube-api-access-xr9ff\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.526489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lwx\" (UniqueName: \"kubernetes.io/projected/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-kube-api-access-c7lwx\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.526489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526445    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-trusted-ca\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.526489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41d46f06-67bd-4895-ad89-91bec576c8dd-ca-trust-extracted\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.526489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-installation-pull-secrets\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.526619 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526512    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.526619 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526560    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-tmp-dir\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.526619 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526587    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-image-registry-private-configuration\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.526619 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-certificates\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.526747 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.526620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-bound-sa-token\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.627790 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.627753    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-tmp-dir\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.627933 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.627801    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-image-registry-private-configuration\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.627933 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.627819    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-certificates\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.627933 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.627837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d878d54-d241-49e1-ac1c-817462ac0e17-cert\") pod \"ingress-canary-v76p8\" (UID: \"2d878d54-d241-49e1-ac1c-817462ac0e17\") " pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:00.628072 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-bound-sa-token\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.628072 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-config-volume\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.628072 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-tmp-dir\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9ff\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-kube-api-access-xr9ff\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628158    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lwx\" (UniqueName: \"kubernetes.io/projected/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-kube-api-access-c7lwx\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628189    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-trusted-ca\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.628202    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qkt\" (UniqueName: \"kubernetes.io/projected/2d878d54-d241-49e1-ac1c-817462ac0e17-kube-api-access-s2qkt\") pod \"ingress-canary-v76p8\" (UID: \"2d878d54-d241-49e1-ac1c-817462ac0e17\") " pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.628223    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767c4f9cc7-gqnqw: secret "image-registry-tls" not found
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41d46f06-67bd-4895-ad89-91bec576c8dd-ca-trust-extracted\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-installation-pull-secrets\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.628294    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls podName:41d46f06-67bd-4895-ad89-91bec576c8dd nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.128272963 +0000 UTC m=+37.121437111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls") pod "image-registry-767c4f9cc7-gqnqw" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd") : secret "image-registry-tls" not found
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.628450    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:00.628489    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls podName:994e4bb0-4168-40b6-9a28-f73dcda6c9a0 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.128475921 +0000 UTC m=+37.121640073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls") pod "dns-default-xshcr" (UID: "994e4bb0-4168-40b6-9a28-f73dcda6c9a0") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628536    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-certificates\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-config-volume\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.628793 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.628801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41d46f06-67bd-4895-ad89-91bec576c8dd-ca-trust-extracted\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.629443 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.629081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-trusted-ca\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.631797 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.631776    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-image-registry-private-configuration\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.631913 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.631896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-installation-pull-secrets\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.639310 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.639290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9ff\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-kube-api-access-xr9ff\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.639781 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.639766    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-bound-sa-token\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:00.643016 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.642943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lwx\" (UniqueName: \"kubernetes.io/projected/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-kube-api-access-c7lwx\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:00.729028 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.728998    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d878d54-d241-49e1-ac1c-817462ac0e17-cert\") pod \"ingress-canary-v76p8\" (UID: \"2d878d54-d241-49e1-ac1c-817462ac0e17\") " pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:00.729157 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.729076    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qkt\" (UniqueName: \"kubernetes.io/projected/2d878d54-d241-49e1-ac1c-817462ac0e17-kube-api-access-s2qkt\") pod \"ingress-canary-v76p8\" (UID: \"2d878d54-d241-49e1-ac1c-817462ac0e17\") " pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:00.731326 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.731305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d878d54-d241-49e1-ac1c-817462ac0e17-cert\") pod \"ingress-canary-v76p8\" (UID: \"2d878d54-d241-49e1-ac1c-817462ac0e17\") " pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:00.733518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.733498    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e69ca4a-a391-454f-a7d1-129e509ef711" containerID="6c937b4acd1c073bf09b07b655e11bda320e5f278d3a64beccb4860c6f444959" exitCode=0
Jul 16 21:48:00.733602 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.733526    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerDied","Data":"6c937b4acd1c073bf09b07b655e11bda320e5f278d3a64beccb4860c6f444959"}
Jul 16 21:48:00.741212 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.741185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qkt\" (UniqueName: \"kubernetes.io/projected/2d878d54-d241-49e1-ac1c-817462ac0e17-kube-api-access-s2qkt\") pod \"ingress-canary-v76p8\" (UID: \"2d878d54-d241-49e1-ac1c-817462ac0e17\") " pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:00.784029 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:00.784006    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v76p8"
Jul 16 21:48:01.013113 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:01.012934    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v76p8"]
Jul 16 21:48:01.015685 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:01.015661    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d878d54_d241_49e1_ac1c_817462ac0e17.slice/crio-4a8df226a61d46d159248f37d82fe29f05e51cd70da6c0337f3056e43ccc1525 WatchSource:0}: Error finding container 4a8df226a61d46d159248f37d82fe29f05e51cd70da6c0337f3056e43ccc1525: Status 404 returned error can't find the container with id 4a8df226a61d46d159248f37d82fe29f05e51cd70da6c0337f3056e43ccc1525
Jul 16 21:48:01.132417 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:01.132333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:01.132417 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:01.132406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:01.132593 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:01.132499    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:01.132593 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:01.132514    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767c4f9cc7-gqnqw: secret "image-registry-tls" not found
Jul 16 21:48:01.132593 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:01.132524    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:01.132593 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:01.132572    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls podName:994e4bb0-4168-40b6-9a28-f73dcda6c9a0 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:02.132558093 +0000 UTC m=+38.125722239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls") pod "dns-default-xshcr" (UID: "994e4bb0-4168-40b6-9a28-f73dcda6c9a0") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:01.132593 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:01.132586    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls podName:41d46f06-67bd-4895-ad89-91bec576c8dd nodeName:}" failed. No retries permitted until 2026-07-16 21:48:02.132580523 +0000 UTC m=+38.125744652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls") pod "image-registry-767c4f9cc7-gqnqw" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd") : secret "image-registry-tls" not found
Jul 16 21:48:01.751352 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:01.751316    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e69ca4a-a391-454f-a7d1-129e509ef711" containerID="515b256f8c2c117837a2138d83c0d382370cffb886797221f2e2192c97b25c5e" exitCode=0
Jul 16 21:48:01.751970 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:01.751404    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerDied","Data":"515b256f8c2c117837a2138d83c0d382370cffb886797221f2e2192c97b25c5e"}
Jul 16 21:48:01.752885 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:01.752856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v76p8" event={"ID":"2d878d54-d241-49e1-ac1c-817462ac0e17","Type":"ContainerStarted","Data":"4a8df226a61d46d159248f37d82fe29f05e51cd70da6c0337f3056e43ccc1525"}
Jul 16 21:48:02.141639 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:02.141599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:02.141812 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:02.141707    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:02.141812 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:02.141790    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:02.141812 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:02.141808    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:02.141941 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:02.141815    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767c4f9cc7-gqnqw: secret "image-registry-tls" not found
Jul 16 21:48:02.141941 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:02.141868    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls podName:994e4bb0-4168-40b6-9a28-f73dcda6c9a0 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:04.141848363 +0000 UTC m=+40.135012518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls") pod "dns-default-xshcr" (UID: "994e4bb0-4168-40b6-9a28-f73dcda6c9a0") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:02.141941 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:02.141885    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls podName:41d46f06-67bd-4895-ad89-91bec576c8dd nodeName:}" failed. No retries permitted until 2026-07-16 21:48:04.141878391 +0000 UTC m=+40.135042521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls") pod "image-registry-767c4f9cc7-gqnqw" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd") : secret "image-registry-tls" not found
Jul 16 21:48:02.758857 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:02.758820    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" event={"ID":"8e69ca4a-a391-454f-a7d1-129e509ef711","Type":"ContainerStarted","Data":"d10a67116e356c9aa201b4727e3f1e217912532dc4fe63d7f05bf0d6fd151b41"}
Jul 16 21:48:02.802795 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:02.802732    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bt8gz" podStartSLOduration=10.118323191 podStartE2EDuration="38.802710879s" totalImagesPullingTime="28.684387688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:02.799580802 +0000 UTC m=+38.792744954" watchObservedRunningTime="2026-07-16 21:48:02.802710879 +0000 UTC m=+38.795875033"
Jul 16 21:48:04.158043 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:04.157806    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:04.158456 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:04.158104    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:04.158456 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:04.157981    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:04.158456 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:04.158210    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767c4f9cc7-gqnqw: secret "image-registry-tls" not found
Jul 16 21:48:04.158456 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:04.158274    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls podName:41d46f06-67bd-4895-ad89-91bec576c8dd nodeName:}" failed. No retries permitted until 2026-07-16 21:48:08.158256107 +0000 UTC m=+44.151420236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls") pod "image-registry-767c4f9cc7-gqnqw" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd") : secret "image-registry-tls" not found
Jul 16 21:48:04.158456 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:04.158170    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:04.158456 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:04.158354    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls podName:994e4bb0-4168-40b6-9a28-f73dcda6c9a0 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:08.158341483 +0000 UTC m=+44.151505613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls") pod "dns-default-xshcr" (UID: "994e4bb0-4168-40b6-9a28-f73dcda6c9a0") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:04.765063 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:04.765028    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v76p8" event={"ID":"2d878d54-d241-49e1-ac1c-817462ac0e17","Type":"ContainerStarted","Data":"c985e1d23a97e3ac654b6266fce062d5697709c24edaf679489c22b91f750843"}
Jul 16 21:48:08.189253 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.189217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:08.189787 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.189284    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:08.189787 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.189365    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:08.189787 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.189386    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767c4f9cc7-gqnqw: secret "image-registry-tls" not found
Jul 16 21:48:08.189787 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.189402    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:08.189787 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.189440    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls podName:41d46f06-67bd-4895-ad89-91bec576c8dd nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.189424851 +0000 UTC m=+52.182588981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls") pod "image-registry-767c4f9cc7-gqnqw" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd") : secret "image-registry-tls" not found
Jul 16 21:48:08.189787 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.189452    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls podName:994e4bb0-4168-40b6-9a28-f73dcda6c9a0 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.189446552 +0000 UTC m=+52.182610681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls") pod "dns-default-xshcr" (UID: "994e4bb0-4168-40b6-9a28-f73dcda6c9a0") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:08.634302 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.634240    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v76p8" podStartSLOduration=5.863801569 podStartE2EDuration="8.634220815s" totalImagesPullingTime="2.770419246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:04.786714146 +0000 UTC m=+40.779878301" watchObservedRunningTime="2026-07-16 21:48:08.634220815 +0000 UTC m=+44.627385169"
Jul 16 21:48:08.634578 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.634566    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-89z2g"]
Jul 16 21:48:08.658710 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.658685    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.662123 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.662101    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:48:08.662220 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.662157    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bcp4l\""
Jul 16 21:48:08.662617 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.662596    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:48:08.662730 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.662622    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:48:08.662730 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.662660    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:48:08.670458 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.670437    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89z2g"]
Jul 16 21:48:08.795545 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.795517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-crio-socket\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.795731 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.795552    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.795731 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.795576    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zzs\" (UniqueName: \"kubernetes.io/projected/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-api-access-x2zzs\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.795731 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.795595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-data-volume\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.795846 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.795732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896268 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.896186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-crio-socket\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896268 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.896220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896268 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.896239    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zzs\" (UniqueName: \"kubernetes.io/projected/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-api-access-x2zzs\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896536 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.896317    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:08.896536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.896365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-data-volume\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896536 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.896382    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:09.396366971 +0000 UTC m=+45.389531101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:08.896536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.896399    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-crio-socket\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.896503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896771 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.896658    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:08.896771 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.896686    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-data-volume\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:08.896771 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:08.896733    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:09.396713501 +0000 UTC m=+45.389877634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:08.907969 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:08.907941    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zzs\" (UniqueName: \"kubernetes.io/projected/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-api-access-x2zzs\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:09.400460 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:09.400424    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:09.400836 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:09.400507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:09.400836 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:09.400555    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:09.400836 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:09.400602    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:09.400836 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:09.400619    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:10.400602947 +0000 UTC m=+46.393767076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:09.400836 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:09.400650    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:10.400639692 +0000 UTC m=+46.393803821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:10.407267 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:10.407235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:10.407630 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:10.407289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:10.407630 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:10.407362    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:10.407630 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:10.407400    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:10.407630 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:10.407413    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:12.407399098 +0000 UTC m=+48.400563228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:10.407630 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:10.407468    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:12.40745033 +0000 UTC m=+48.400614463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:10.586179 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:10.586147    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rxtx7_86f80ab0-9e2d-4881-b1c1-71fd4a113f8e/dns-node-resolver/0.log"
Jul 16 21:48:11.076555 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.076524    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"]
Jul 16 21:48:11.124331 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.124300    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"]
Jul 16 21:48:11.124476 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.124393    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:11.127926 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.127893    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ddm8s\""
Jul 16 21:48:11.128068 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.127924    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 16 21:48:11.128896 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.128875    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 16 21:48:11.316363 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.316327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:11.316549 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.316385    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:11.417534 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.417453    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:11.418001 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.417558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:11.418001 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:11.417664    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:48:11.418001 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:11.417715    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert podName:49044b9b-aac9-4ff6-ac8b-b10a40b841f6 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:11.91770207 +0000 UTC m=+47.910866199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-hlgk5" (UID: "49044b9b-aac9-4ff6-ac8b-b10a40b841f6") : secret "networking-console-plugin-cert" not found
Jul 16 21:48:11.418302 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.418277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:11.596474 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.596449    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8rmsf_73baff7b-e485-492f-97da-09da89b0cc69/node-ca/0.log"
Jul 16 21:48:11.921788 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:11.921754    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:11.921974 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:11.921903    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:48:11.922030 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:11.921983    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert podName:49044b9b-aac9-4ff6-ac8b-b10a40b841f6 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:12.921952008 +0000 UTC m=+48.915116138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-hlgk5" (UID: "49044b9b-aac9-4ff6-ac8b-b10a40b841f6") : secret "networking-console-plugin-cert" not found
Jul 16 21:48:12.425928 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:12.425893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:12.426377 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:12.425976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:12.426377 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:12.426050    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:12.426377 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:12.426117    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.426098593 +0000 UTC m=+52.419262722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:12.426377 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:12.426141    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:12.426377 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:12.426209    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.426190835 +0000 UTC m=+52.419354965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:12.588547 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:12.588512    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v76p8_2d878d54-d241-49e1-ac1c-817462ac0e17/serve-healthcheck-canary/0.log"
Jul 16 21:48:12.928600 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:12.928562    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:12.928774 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:12.928703    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:48:12.928774 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:12.928766    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert podName:49044b9b-aac9-4ff6-ac8b-b10a40b841f6 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:14.928748673 +0000 UTC m=+50.921912825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-hlgk5" (UID: "49044b9b-aac9-4ff6-ac8b-b10a40b841f6") : secret "networking-console-plugin-cert" not found
Jul 16 21:48:14.946369 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:14.946333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:14.946818 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:14.946484    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:48:14.946818 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:14.946559    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert podName:49044b9b-aac9-4ff6-ac8b-b10a40b841f6 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:18.946542232 +0000 UTC m=+54.939706362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-hlgk5" (UID: "49044b9b-aac9-4ff6-ac8b-b10a40b841f6") : secret "networking-console-plugin-cert" not found
Jul 16 21:48:16.257398 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:16.257352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:16.257855 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:16.257441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:16.257855 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:16.257535    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:16.257855 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:16.257547    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767c4f9cc7-gqnqw: secret "image-registry-tls" not found
Jul 16 21:48:16.257855 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:16.257533    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:16.257855 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:16.257590    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls podName:41d46f06-67bd-4895-ad89-91bec576c8dd nodeName:}" failed. No retries permitted until 2026-07-16 21:48:32.257576667 +0000 UTC m=+68.250740796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls") pod "image-registry-767c4f9cc7-gqnqw" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd") : secret "image-registry-tls" not found
Jul 16 21:48:16.257855 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:16.257637    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls podName:994e4bb0-4168-40b6-9a28-f73dcda6c9a0 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:32.257621182 +0000 UTC m=+68.250785318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls") pod "dns-default-xshcr" (UID: "994e4bb0-4168-40b6-9a28-f73dcda6c9a0") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:16.459685 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:16.459646    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:16.459799 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:16.459714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:16.459853 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:16.459818    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:16.459912 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:16.459898    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls podName:cbe6c5ec-9f02-4dfe-b33f-c47f12527380 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:24.45987819 +0000 UTC m=+60.453042323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89z2g" (UID: "cbe6c5ec-9f02-4dfe-b33f-c47f12527380") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:16.460252 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:16.460232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:18.979921 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:18.979875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:18.980386 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:18.980041    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:48:18.980386 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:48:18.980111    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert podName:49044b9b-aac9-4ff6-ac8b-b10a40b841f6 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:26.980095105 +0000 UTC m=+62.973259234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-hlgk5" (UID: "49044b9b-aac9-4ff6-ac8b-b10a40b841f6") : secret "networking-console-plugin-cert" not found
Jul 16 21:48:24.520675 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:24.520637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:24.522699 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:24.522681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe6c5ec-9f02-4dfe-b33f-c47f12527380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89z2g\" (UID: \"cbe6c5ec-9f02-4dfe-b33f-c47f12527380\") " pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:24.570924 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:24.570900    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bcp4l\""
Jul 16 21:48:24.578433 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:24.578419    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89z2g"
Jul 16 21:48:24.734311 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:24.734277    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-97jzh"
Jul 16 21:48:24.742725 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:24.742705    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89z2g"]
Jul 16 21:48:24.744586 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:24.744563    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe6c5ec_9f02_4dfe_b33f_c47f12527380.slice/crio-1345bbcc06e7ca1673419dd1a7f5ff7a91d26b93a5971711c17e2ae4226b445b WatchSource:0}: Error finding container 1345bbcc06e7ca1673419dd1a7f5ff7a91d26b93a5971711c17e2ae4226b445b: Status 404 returned error can't find the container with id 1345bbcc06e7ca1673419dd1a7f5ff7a91d26b93a5971711c17e2ae4226b445b
Jul 16 21:48:24.810670 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:24.810591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89z2g" event={"ID":"cbe6c5ec-9f02-4dfe-b33f-c47f12527380","Type":"ContainerStarted","Data":"1345bbcc06e7ca1673419dd1a7f5ff7a91d26b93a5971711c17e2ae4226b445b"}
Jul 16 21:48:27.043690 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:27.043640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:27.046437 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:27.046409    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49044b9b-aac9-4ff6-ac8b-b10a40b841f6-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-hlgk5\" (UID: \"49044b9b-aac9-4ff6-ac8b-b10a40b841f6\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:27.336805 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:27.336776    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ddm8s\""
Jul 16 21:48:27.344613 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:27.344576    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"
Jul 16 21:48:29.598604 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:29.598576    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5"]
Jul 16 21:48:29.601589 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:29.601565    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49044b9b_aac9_4ff6_ac8b_b10a40b841f6.slice/crio-ba45185fc2ca4691cc304cc715085bcf80776aa19b299260260455cf0c7059d6 WatchSource:0}: Error finding container ba45185fc2ca4691cc304cc715085bcf80776aa19b299260260455cf0c7059d6: Status 404 returned error can't find the container with id ba45185fc2ca4691cc304cc715085bcf80776aa19b299260260455cf0c7059d6
Jul 16 21:48:29.823936 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:29.823859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5" event={"ID":"49044b9b-aac9-4ff6-ac8b-b10a40b841f6","Type":"ContainerStarted","Data":"ba45185fc2ca4691cc304cc715085bcf80776aa19b299260260455cf0c7059d6"}
Jul 16 21:48:29.825166 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:29.825142    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89z2g" event={"ID":"cbe6c5ec-9f02-4dfe-b33f-c47f12527380","Type":"ContainerStarted","Data":"856c4e20b1d4db38c3f0c7a4db43507f929b9116464d001dd7ed6319c324ea42"}
Jul 16 21:48:32.289333 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.289242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:48:32.289333 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.289306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:32.289333 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.289335    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:48:32.289889 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.289391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:32.292566 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.292538    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/994e4bb0-4168-40b6-9a28-f73dcda6c9a0-metrics-tls\") pod \"dns-default-xshcr\" (UID: \"994e4bb0-4168-40b6-9a28-f73dcda6c9a0\") " pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:32.292711 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.292639    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"image-registry-767c4f9cc7-gqnqw\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") " pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:32.293837 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.293813    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:48:32.295326 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.295301    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:48:32.303345 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.303317    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/186438be-cc8d-4091-897c-f83f9acd1c5c-original-pull-secret\") pod \"global-pull-secret-syncer-c4l4q\" (UID: \"186438be-cc8d-4091-897c-f83f9acd1c5c\") " pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:48:32.304547 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.304256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e18395e-950a-4347-a71c-e91f9bd7fc70-metrics-certs\") pod \"network-metrics-daemon-7xsb4\" (UID: \"1e18395e-950a-4347-a71c-e91f9bd7fc70\") " pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:48:32.304547 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.304365    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c4l4q"
Jul 16 21:48:32.323031 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.321925    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6g595\""
Jul 16 21:48:32.329434 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.329409    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xsb4"
Jul 16 21:48:32.390766 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.390734    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:48:32.399762 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.399736    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:48:32.405229 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.405207    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:48:32.413641 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.413614    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"]
Jul 16 21:48:32.414589 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.414534    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84m47\" (UniqueName: \"kubernetes.io/projected/46aaf5d5-4bb7-4fa0-ba40-47bc5329139b-kube-api-access-84m47\") pod \"network-check-target-6t5qm\" (UID: \"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b\") " pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:48:32.418046 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.418024    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"]
Jul 16 21:48:32.418203 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.418186    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.420438 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.420415    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.421791 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.421762    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 16 21:48:32.422033 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.422010    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 21:48:32.422120 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.422064    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 21:48:32.423670 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.423653    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 21:48:32.427153 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.427102    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 21:48:32.427575 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.427460    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 21:48:32.427776 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.427755    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 21:48:32.427886 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.427868    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 21:48:32.436460 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.436436    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"]
Jul 16 21:48:32.456012 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.455984    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"]
Jul 16 21:48:32.467414 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.467383    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-767c4f9cc7-gqnqw"]
Jul 16 21:48:32.470643 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.470621    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdfjl\""
Jul 16 21:48:32.477701 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.477678    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:32.492358 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97631eee-27da-4a95-b27c-bcc3a1ee880c-tmp\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.492471 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-ca\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.492471 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492423    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7ba56-f906-4de6-9439-c9a0261bd305-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.492471 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv98b\" (UniqueName: \"kubernetes.io/projected/97631eee-27da-4a95-b27c-bcc3a1ee880c-kube-api-access-sv98b\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.492585 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6hp\" (UniqueName: \"kubernetes.io/projected/3fa7ba56-f906-4de6-9439-c9a0261bd305-kube-api-access-kg6hp\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.492585 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492525    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.492585 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97631eee-27da-4a95-b27c-bcc3a1ee880c-klusterlet-config\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.492585 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492579    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-hub\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.492718 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.492618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.579180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.579149    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9tv9m\""
Jul 16 21:48:32.583083 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.583062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:32.593750 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.593729    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6hp\" (UniqueName: \"kubernetes.io/projected/3fa7ba56-f906-4de6-9439-c9a0261bd305-kube-api-access-kg6hp\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.593868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.593760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.593868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.593780    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97631eee-27da-4a95-b27c-bcc3a1ee880c-klusterlet-config\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.593868 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.593805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-hub\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.594320 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.594292    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.594421 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.594372    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97631eee-27da-4a95-b27c-bcc3a1ee880c-tmp\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.594421 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.594408    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-ca\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.594545 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.594450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7ba56-f906-4de6-9439-c9a0261bd305-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.594545 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.594477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv98b\" (UniqueName: \"kubernetes.io/projected/97631eee-27da-4a95-b27c-bcc3a1ee880c-kube-api-access-sv98b\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.594913 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.594786    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97631eee-27da-4a95-b27c-bcc3a1ee880c-tmp\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.595452 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.595426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7ba56-f906-4de6-9439-c9a0261bd305-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.596548 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.596529    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97631eee-27da-4a95-b27c-bcc3a1ee880c-klusterlet-config\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.596650 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.596585    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.596650 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.596628    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-hub\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.597170 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.597143    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-ca\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.597257 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.597205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fa7ba56-f906-4de6-9439-c9a0261bd305-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.601509 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.601491    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mc6l9\""
Jul 16 21:48:32.609867 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.609846    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:48:32.619898 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.619877    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6hp\" (UniqueName: \"kubernetes.io/projected/3fa7ba56-f906-4de6-9439-c9a0261bd305-kube-api-access-kg6hp\") pod \"cluster-proxy-proxy-agent-68ff6c9d87-vw5x6\" (UID: \"3fa7ba56-f906-4de6-9439-c9a0261bd305\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:32.625559 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.625536    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv98b\" (UniqueName: \"kubernetes.io/projected/97631eee-27da-4a95-b27c-bcc3a1ee880c-kube-api-access-sv98b\") pod \"klusterlet-addon-workmgr-58898dbcd-8wgmw\" (UID: \"97631eee-27da-4a95-b27c-bcc3a1ee880c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.731542 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.731512    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:32.744974 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:32.744928    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"
Jul 16 21:48:33.336528 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.336433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xshcr"]
Jul 16 21:48:33.549522 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.549486    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6"]
Jul 16 21:48:33.553552 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:33.553495    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fa7ba56_f906_4de6_9439_c9a0261bd305.slice/crio-3284ccd60a0c612b8b88a5228d1f4b1b91711a81825a67405f5ae65aa3f45992 WatchSource:0}: Error finding container 3284ccd60a0c612b8b88a5228d1f4b1b91711a81825a67405f5ae65aa3f45992: Status 404 returned error can't find the container with id 3284ccd60a0c612b8b88a5228d1f4b1b91711a81825a67405f5ae65aa3f45992
Jul 16 21:48:33.555923 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.555890    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"]
Jul 16 21:48:33.557074 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:33.557054    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97631eee_27da_4a95_b27c_bcc3a1ee880c.slice/crio-755d91b4be5fc2815ed925aac5e04c1283179c20b87fd5756c3203a144e77508 WatchSource:0}: Error finding container 755d91b4be5fc2815ed925aac5e04c1283179c20b87fd5756c3203a144e77508: Status 404 returned error can't find the container with id 755d91b4be5fc2815ed925aac5e04c1283179c20b87fd5756c3203a144e77508
Jul 16 21:48:33.764317 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.764236    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6t5qm"]
Jul 16 21:48:33.768159 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:33.768040    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aaf5d5_4bb7_4fa0_ba40_47bc5329139b.slice/crio-df5620996293916da0c8e98b9b7e071de5cb4125b60c345e30bf4ddbbbe82633 WatchSource:0}: Error finding container df5620996293916da0c8e98b9b7e071de5cb4125b60c345e30bf4ddbbbe82633: Status 404 returned error can't find the container with id df5620996293916da0c8e98b9b7e071de5cb4125b60c345e30bf4ddbbbe82633
Jul 16 21:48:33.770218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.770196    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c4l4q"]
Jul 16 21:48:33.770936 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.770912    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7xsb4"]
Jul 16 21:48:33.775129 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.775049    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-767c4f9cc7-gqnqw"]
Jul 16 21:48:33.775492 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:33.775454    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e18395e_950a_4347_a71c_e91f9bd7fc70.slice/crio-8658041dd8772452c26021583840c4c56aef31fca989d6c115b9ecbc28fc00e2 WatchSource:0}: Error finding container 8658041dd8772452c26021583840c4c56aef31fca989d6c115b9ecbc28fc00e2: Status 404 returned error can't find the container with id 8658041dd8772452c26021583840c4c56aef31fca989d6c115b9ecbc28fc00e2
Jul 16 21:48:33.838432 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.838395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5" event={"ID":"49044b9b-aac9-4ff6-ac8b-b10a40b841f6","Type":"ContainerStarted","Data":"d6b0153cbb1a27df36258c20e6ffe34f1f982e2afd5386b5c0f352e047214455"}
Jul 16 21:48:33.839926 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.839861    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6t5qm" event={"ID":"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b","Type":"ContainerStarted","Data":"df5620996293916da0c8e98b9b7e071de5cb4125b60c345e30bf4ddbbbe82633"}
Jul 16 21:48:33.841219 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.841194    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6" event={"ID":"3fa7ba56-f906-4de6-9439-c9a0261bd305","Type":"ContainerStarted","Data":"3284ccd60a0c612b8b88a5228d1f4b1b91711a81825a67405f5ae65aa3f45992"}
Jul 16 21:48:33.842571 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.842540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c4l4q" event={"ID":"186438be-cc8d-4091-897c-f83f9acd1c5c","Type":"ContainerStarted","Data":"8d667d309e6064ad3f13ceece68144ab9c8d183dc2916d2d211285debac6227d"}
Jul 16 21:48:33.843931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.843907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw" event={"ID":"41d46f06-67bd-4895-ad89-91bec576c8dd","Type":"ContainerStarted","Data":"f338606b041d5ba6bb009ce02c2320993cdfc406e7f1d5ac2dc3dcbce3eab04a"}
Jul 16 21:48:33.845910 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.845886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89z2g" event={"ID":"cbe6c5ec-9f02-4dfe-b33f-c47f12527380","Type":"ContainerStarted","Data":"d261675592658758c72070586a6a4989b0df6dd6ed831a3ccc2f0b7c663e8218"}
Jul 16 21:48:33.847392 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.847353    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw" event={"ID":"97631eee-27da-4a95-b27c-bcc3a1ee880c","Type":"ContainerStarted","Data":"755d91b4be5fc2815ed925aac5e04c1283179c20b87fd5756c3203a144e77508"}
Jul 16 21:48:33.848760 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.848700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7xsb4" event={"ID":"1e18395e-950a-4347-a71c-e91f9bd7fc70","Type":"ContainerStarted","Data":"8658041dd8772452c26021583840c4c56aef31fca989d6c115b9ecbc28fc00e2"}
Jul 16 21:48:33.850012 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.849990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xshcr" event={"ID":"994e4bb0-4168-40b6-9a28-f73dcda6c9a0","Type":"ContainerStarted","Data":"ad05b9f6d60d2a6c9ea71e67003a9bf1c503ae6c1da9674d26a70a63642262f0"}
Jul 16 21:48:33.859536 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:33.859043    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-hlgk5" podStartSLOduration=19.271869972 podStartE2EDuration="22.859024998s" totalImagesPullingTime="3.587155026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:33.857784995 +0000 UTC m=+69.850949158" watchObservedRunningTime="2026-07-16 21:48:33.859024998 +0000 UTC m=+69.852189143"
Jul 16 21:48:34.863150 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:34.863113    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw" event={"ID":"41d46f06-67bd-4895-ad89-91bec576c8dd","Type":"ContainerStarted","Data":"edf22dee69d0cd5b78d4d06093995fe629d054842a34fe206ff3b23829901bc4"}
Jul 16 21:48:34.863630 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:34.863194    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:34.892867 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:34.892578    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw" podStartSLOduration=37.892556524 podStartE2EDuration="37.892556524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:34.890765038 +0000 UTC m=+70.883929191" watchObservedRunningTime="2026-07-16 21:48:34.892556524 +0000 UTC m=+70.885720677"
Jul 16 21:48:44.902126 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.902089    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89z2g" event={"ID":"cbe6c5ec-9f02-4dfe-b33f-c47f12527380","Type":"ContainerStarted","Data":"50f8be81cc243da9d21a73ce951abc9f5b5323c660cce392a9a8e5376c1fd099"}
Jul 16 21:48:44.903599 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.903576    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw" event={"ID":"97631eee-27da-4a95-b27c-bcc3a1ee880c","Type":"ContainerStarted","Data":"277d7bb0afc77fae1d8fe25beb773a26d6bd1945ef669d1329f30eb332dcc0b8"}
Jul 16 21:48:44.903705 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.903624    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:44.905558 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.905517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7xsb4" event={"ID":"1e18395e-950a-4347-a71c-e91f9bd7fc70","Type":"ContainerStarted","Data":"1bccd58b8481ba88a49180b22fd8b568d19e2d3a86515016e3a7be12edccd25f"}
Jul 16 21:48:44.905687 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.905566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7xsb4" event={"ID":"1e18395e-950a-4347-a71c-e91f9bd7fc70","Type":"ContainerStarted","Data":"9d909f5198b7d0199afd7cff3719a8b921ed40928219460d9dff063553945268"}
Jul 16 21:48:44.905853 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.905835    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw"
Jul 16 21:48:44.907368 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.907339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xshcr" event={"ID":"994e4bb0-4168-40b6-9a28-f73dcda6c9a0","Type":"ContainerStarted","Data":"e75f099f40df444c66fa394525eee723207a0c50fb67a6f3bcdf86821234babf"}
Jul 16 21:48:44.907466 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.907386    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xshcr" event={"ID":"994e4bb0-4168-40b6-9a28-f73dcda6c9a0","Type":"ContainerStarted","Data":"ec57077a5ad096f67e17a53f5defb671f5de15c5893f02790a46595f8ce3846b"}
Jul 16 21:48:44.907466 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.907449    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:44.908685 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.908662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6t5qm" event={"ID":"46aaf5d5-4bb7-4fa0-ba40-47bc5329139b","Type":"ContainerStarted","Data":"d07641f1cafdfe7f20cb65b7a22e0ee56a4056703ca268a55a832cbafdfd843b"}
Jul 16 21:48:44.908767 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.908722    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:48:44.910562 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.910537    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6" event={"ID":"3fa7ba56-f906-4de6-9439-c9a0261bd305","Type":"ContainerStarted","Data":"7b60925f5a10b22c69cedcbf56b6182bf5478bf8a8f3e8983ec82e317f522327"}
Jul 16 21:48:44.910655 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.910570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6" event={"ID":"3fa7ba56-f906-4de6-9439-c9a0261bd305","Type":"ContainerStarted","Data":"0731eb00342df145c55ada28faf500cd710269cf2501721650340fc0fbcad43c"}
Jul 16 21:48:44.910655 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.910587    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6" event={"ID":"3fa7ba56-f906-4de6-9439-c9a0261bd305","Type":"ContainerStarted","Data":"15c36a3ed93022bce376d18d7f9bc7dbf28bafb665518fb96d0a49d9cd80a6c2"}
Jul 16 21:48:44.911829 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.911808    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c4l4q" event={"ID":"186438be-cc8d-4091-897c-f83f9acd1c5c","Type":"ContainerStarted","Data":"6f58ea3a6273eed4c1ccda73267068be5160c48f285356209cb0f02a76121066"}
Jul 16 21:48:44.930548 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.930500    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-89z2g" podStartSLOduration=17.361204079 podStartE2EDuration="36.93048485s" totalImagesPullingTime="19.569280771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:48:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:44.929895555 +0000 UTC m=+80.923059709" watchObservedRunningTime="2026-07-16 21:48:44.93048485 +0000 UTC m=+80.923649001"
Jul 16 21:48:44.953684 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.953590    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58898dbcd-8wgmw" podStartSLOduration=2.027866638 podStartE2EDuration="12.953573748s" totalImagesPullingTime="10.92570711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:44.953244452 +0000 UTC m=+80.946408617" watchObservedRunningTime="2026-07-16 21:48:44.953573748 +0000 UTC m=+80.946737899"
Jul 16 21:48:44.987797 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:44.987730    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68ff6c9d87-vw5x6" podStartSLOduration=2.059516492 podStartE2EDuration="12.987709759s" totalImagesPullingTime="10.928193267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:44.987039911 +0000 UTC m=+80.980204063" watchObservedRunningTime="2026-07-16 21:48:44.987709759 +0000 UTC m=+80.980873912"
Jul 16 21:48:45.010093 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:45.010047    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c4l4q" podStartSLOduration=70.270693401 podStartE2EDuration="1m21.010032721s" totalImagesPullingTime="10.73933932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:45.009214651 +0000 UTC m=+81.002378800" watchObservedRunningTime="2026-07-16 21:48:45.010032721 +0000 UTC m=+81.003196870"
Jul 16 21:48:45.035720 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:45.035670    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7xsb4" podStartSLOduration=70.389475507 podStartE2EDuration="1m21.035655969s" totalImagesPullingTime="10.646180462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:45.035354069 +0000 UTC m=+81.028518222" watchObservedRunningTime="2026-07-16 21:48:45.035655969 +0000 UTC m=+81.028820120"
Jul 16 21:48:45.061216 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:45.061159    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6t5qm" podStartSLOduration=70.355732859 podStartE2EDuration="1m21.061138378s" totalImagesPullingTime="10.705405519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:45.060496033 +0000 UTC m=+81.053660187" watchObservedRunningTime="2026-07-16 21:48:45.061138378 +0000 UTC m=+81.054302527"
Jul 16 21:48:45.087030 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:45.086975    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xshcr" podStartSLOduration=34.013291113 podStartE2EDuration="45.086940823s" totalImagesPullingTime="11.07364971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:45.085199511 +0000 UTC m=+81.078363666" watchObservedRunningTime="2026-07-16 21:48:45.086940823 +0000 UTC m=+81.080105015"
Jul 16 21:48:48.984992 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.984863    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kxhgh"]
Jul 16 21:48:48.989280 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.989261    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:48.993994 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.993950    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:48:48.994186 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.994170    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:48:48.995389 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.995367    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8w4k\""
Jul 16 21:48:48.995529 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.995367    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:48:48.995529 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.995372    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:48:48.995529 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.995375    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:48:48.997418 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:48.997401    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:48:49.137839 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.137797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-sys\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.137839 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.137845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-textfile\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.138102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.137873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqzs\" (UniqueName: \"kubernetes.io/projected/90df56fe-e157-48f7-8303-9e5780fa8ce8-kube-api-access-lhqzs\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.138102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.137940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.138102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.137995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-tls\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.138102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.138054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-root\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.138102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.138094    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90df56fe-e157-48f7-8303-9e5780fa8ce8-metrics-client-ca\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.138319 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.138144    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-wtmp\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.138319 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.138172    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239214 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239112    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-textfile\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239214 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239156    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqzs\" (UniqueName: \"kubernetes.io/projected/90df56fe-e157-48f7-8303-9e5780fa8ce8-kube-api-access-lhqzs\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239214 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239185    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239483 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-tls\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239483 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-root\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239483 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239384    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90df56fe-e157-48f7-8303-9e5780fa8ce8-metrics-client-ca\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239483 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-root\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239483 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-wtmp\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239757 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239757 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239537    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-sys\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239757 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239559    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-wtmp\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239757 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239584    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-textfile\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.239757 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239609    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90df56fe-e157-48f7-8303-9e5780fa8ce8-sys\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.240033 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.239984    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90df56fe-e157-48f7-8303-9e5780fa8ce8-metrics-client-ca\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.240118 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.240088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.241581 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.241559    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-tls\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.241643 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.241589    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90df56fe-e157-48f7-8303-9e5780fa8ce8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.250112 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.250089    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqzs\" (UniqueName: \"kubernetes.io/projected/90df56fe-e157-48f7-8303-9e5780fa8ce8-kube-api-access-lhqzs\") pod \"node-exporter-kxhgh\" (UID: \"90df56fe-e157-48f7-8303-9e5780fa8ce8\") " pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.298751 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.298719    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxhgh"
Jul 16 21:48:49.310224 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:49.310199    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90df56fe_e157_48f7_8303_9e5780fa8ce8.slice/crio-7ce6799bc784c3d04509f4b9ca667ca7069e3198dc6c600240fafb7b360324ff WatchSource:0}: Error finding container 7ce6799bc784c3d04509f4b9ca667ca7069e3198dc6c600240fafb7b360324ff: Status 404 returned error can't find the container with id 7ce6799bc784c3d04509f4b9ca667ca7069e3198dc6c600240fafb7b360324ff
Jul 16 21:48:49.930541 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:49.930509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxhgh" event={"ID":"90df56fe-e157-48f7-8303-9e5780fa8ce8","Type":"ContainerStarted","Data":"7ce6799bc784c3d04509f4b9ca667ca7069e3198dc6c600240fafb7b360324ff"}
Jul 16 21:48:50.935634 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:50.935596    2555 generic.go:356] "Generic (PLEG): container finished" podID="90df56fe-e157-48f7-8303-9e5780fa8ce8" containerID="332ed2c8a54f9ce625906b44515b9b9dd2bf8f484ae1d497a6b547bb1079a0c1" exitCode=0
Jul 16 21:48:50.936011 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:50.935673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxhgh" event={"ID":"90df56fe-e157-48f7-8303-9e5780fa8ce8","Type":"ContainerDied","Data":"332ed2c8a54f9ce625906b44515b9b9dd2bf8f484ae1d497a6b547bb1079a0c1"}
Jul 16 21:48:51.941449 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:51.941413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxhgh" event={"ID":"90df56fe-e157-48f7-8303-9e5780fa8ce8","Type":"ContainerStarted","Data":"7845239b0fbcceb6e846e2e6b65c79b99bea6a35a93e8f218c3f66ce42276492"}
Jul 16 21:48:51.941859 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:51.941464    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxhgh" event={"ID":"90df56fe-e157-48f7-8303-9e5780fa8ce8","Type":"ContainerStarted","Data":"ab24b2a7fba0f9ca2e2ea02a881332fc55fe22b1a1d35df105ea9b91ec181764"}
Jul 16 21:48:51.968506 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:51.968462    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kxhgh" podStartSLOduration=2.821193913 podStartE2EDuration="3.968448049s" totalImagesPullingTime="1.071157398s" totalInitContainerRuntime="76.096738ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:51.966944027 +0000 UTC m=+87.960108207" watchObservedRunningTime="2026-07-16 21:48:51.968448049 +0000 UTC m=+87.961612201"
Jul 16 21:48:54.868621 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:54.868588    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:48:54.918535 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:54.918506    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xshcr"
Jul 16 21:48:55.612304 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.612272    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:48:55.742444 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.742408    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:48:55.742598 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.742538    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.766446 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.766417    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 21:48:55.770088 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.769898    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 21:48:55.770088 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.769914    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 21:48:55.770088 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.769943    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 21:48:55.770088 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.769987    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 21:48:55.770088 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770034    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 21:48:55.770088 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.769920    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 21:48:55.770088 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.769920    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 21:48:55.770521 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770133    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 21:48:55.770521 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770142    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-42u5ha48sqvkh\""
Jul 16 21:48:55.770521 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770220    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 21:48:55.770521 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770258    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 21:48:55.770521 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770282    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 21:48:55.770521 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770290    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 21:48:55.770521 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.770289    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zksjq\""
Jul 16 21:48:55.772532 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.772518    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 21:48:55.776848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.776826    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 21:48:55.892766 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.892766 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892727    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.892766 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892761    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config-out\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892853    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892871    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892884    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktqp\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-kube-api-access-bktqp\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.892951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893063    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893141    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893575 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893222    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893575 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893243    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-web-config\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893575 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893263    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.893575 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.893287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994253 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config-out\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994253 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994254    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktqp\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-kube-api-access-bktqp\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994489 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994683 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994683 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994547    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994683 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994574    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994683 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994652    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994902 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994702    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994902 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994902 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994773    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994902 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-web-config\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994902 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994902 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994855    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.994902 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.995271 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.994913    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.995271 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.995086    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.995271 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.995122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.995420 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.995282    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.995420 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.995287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998047 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.997684    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998047 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.997775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998047 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.997906    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998047 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.997984    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998346 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.998144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998346 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.998274    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998462 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.998437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998622 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.998591    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config-out\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998979 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.998760    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.998979 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.998817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.999146 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.998981    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-web-config\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.999311 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.999291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.999396 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.999361    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:55.999988 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:55.999944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:56.000403 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:56.000381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:56.000820 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:56.000804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:56.000860 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:56.000820    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:56.006747 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:56.006723    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktqp\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-kube-api-access-bktqp\") pod \"prometheus-k8s-0\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:56.051964 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:56.051937    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:48:56.232513 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:56.232482    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:48:56.233495 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:48:56.233472    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c221a9_b9b5_4d2d_81b3_59cc1b044238.slice/crio-18c1618cd122a5fdf9086b7244a5bac3044f2bf7c20b4bf2c33d5100194d636c WatchSource:0}: Error finding container 18c1618cd122a5fdf9086b7244a5bac3044f2bf7c20b4bf2c33d5100194d636c: Status 404 returned error can't find the container with id 18c1618cd122a5fdf9086b7244a5bac3044f2bf7c20b4bf2c33d5100194d636c
Jul 16 21:48:56.960855 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:56.960819    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerStarted","Data":"18c1618cd122a5fdf9086b7244a5bac3044f2bf7c20b4bf2c33d5100194d636c"}
Jul 16 21:48:57.965990 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:57.965883    2555 generic.go:356] "Generic (PLEG): container finished" podID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a" exitCode=0
Jul 16 21:48:57.965990 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:57.965931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"}
Jul 16 21:48:59.913413 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:48:59.913338    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw" podUID="41d46f06-67bd-4895-ad89-91bec576c8dd" containerName="registry" containerID="cri-o://edf22dee69d0cd5b78d4d06093995fe629d054842a34fe206ff3b23829901bc4" gracePeriod=30
Jul 16 21:49:00.980182 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:00.980145    2555 generic.go:356] "Generic (PLEG): container finished" podID="41d46f06-67bd-4895-ad89-91bec576c8dd" containerID="edf22dee69d0cd5b78d4d06093995fe629d054842a34fe206ff3b23829901bc4" exitCode=0
Jul 16 21:49:00.980591 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:00.980242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw" event={"ID":"41d46f06-67bd-4895-ad89-91bec576c8dd","Type":"ContainerDied","Data":"edf22dee69d0cd5b78d4d06093995fe629d054842a34fe206ff3b23829901bc4"}
Jul 16 21:49:01.290991 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.290948    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:49:01.349268 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349239    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41d46f06-67bd-4895-ad89-91bec576c8dd-ca-trust-extracted\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.349429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.349429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349306    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr9ff\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-kube-api-access-xr9ff\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.349429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349329    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-image-registry-private-configuration\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.349429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349366    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-trusted-ca\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.349429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349389    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-certificates\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.349429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349410    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-installation-pull-secrets\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.349759 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349433    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-bound-sa-token\") pod \"41d46f06-67bd-4895-ad89-91bec576c8dd\" (UID: \"41d46f06-67bd-4895-ad89-91bec576c8dd\") "
Jul 16 21:49:01.350031 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.349988    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:49:01.350031 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.350004    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:49:01.352059 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.352004    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:49:01.352197 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.352163    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:49:01.352334 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.352305    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-kube-api-access-xr9ff" (OuterVolumeSpecName: "kube-api-access-xr9ff") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "kube-api-access-xr9ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:49:01.352401 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.352367    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:49:01.352466 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.352438    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:49:01.357929 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.357906    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d46f06-67bd-4895-ad89-91bec576c8dd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "41d46f06-67bd-4895-ad89-91bec576c8dd" (UID: "41d46f06-67bd-4895-ad89-91bec576c8dd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:49:01.450888 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450851    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41d46f06-67bd-4895-ad89-91bec576c8dd-ca-trust-extracted\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.450888 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450882    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-tls\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.450888 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450891    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9ff\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-kube-api-access-xr9ff\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.451118 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450900    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-image-registry-private-configuration\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.451118 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450910    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-trusted-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.451118 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450918    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41d46f06-67bd-4895-ad89-91bec576c8dd-registry-certificates\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.451118 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450926    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41d46f06-67bd-4895-ad89-91bec576c8dd-installation-pull-secrets\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.451118 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.450934    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41d46f06-67bd-4895-ad89-91bec576c8dd-bound-sa-token\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:49:01.988275 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.988180    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerStarted","Data":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"}
Jul 16 21:49:01.988678 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.988292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerStarted","Data":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"}
Jul 16 21:49:01.989600 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.989578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw" event={"ID":"41d46f06-67bd-4895-ad89-91bec576c8dd","Type":"ContainerDied","Data":"f338606b041d5ba6bb009ce02c2320993cdfc406e7f1d5ac2dc3dcbce3eab04a"}
Jul 16 21:49:01.989705 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.989608    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767c4f9cc7-gqnqw"
Jul 16 21:49:01.989705 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:01.989643    2555 scope.go:117] "RemoveContainer" containerID="edf22dee69d0cd5b78d4d06093995fe629d054842a34fe206ff3b23829901bc4"
Jul 16 21:49:02.024726 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:02.024698    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-767c4f9cc7-gqnqw"]
Jul 16 21:49:02.030185 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:02.030085    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-767c4f9cc7-gqnqw"]
Jul 16 21:49:02.495449 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:02.495413    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d46f06-67bd-4895-ad89-91bec576c8dd" path="/var/lib/kubelet/pods/41d46f06-67bd-4895-ad89-91bec576c8dd/volumes"
Jul 16 21:49:05.003816 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:05.003731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerStarted","Data":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"}
Jul 16 21:49:05.004203 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:05.003829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerStarted","Data":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"}
Jul 16 21:49:05.004203 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:05.003856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerStarted","Data":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"}
Jul 16 21:49:05.004203 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:05.003881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerStarted","Data":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"}
Jul 16 21:49:05.078295 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:05.078214    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.581135962 podStartE2EDuration="10.078197309s" totalImagesPullingTime="7.468541546s" totalInitContainerRuntime="28.519801ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:05.075453892 +0000 UTC m=+101.068618044" watchObservedRunningTime="2026-07-16 21:49:05.078197309 +0000 UTC m=+101.071361454"
Jul 16 21:49:06.052798 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:06.052761    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:49:15.918772 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:15.918741    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6t5qm"
Jul 16 21:49:56.053150 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:56.053110    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:49:56.071847 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:56.071823    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:49:56.190593 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:49:56.190567    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:14.158503 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.158460    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:50:14.159033 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.158873    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="prometheus" containerID="cri-o://27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84" gracePeriod=600
Jul 16 21:50:14.159033 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.158897    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy" containerID="cri-o://cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e" gracePeriod=600
Jul 16 21:50:14.159033 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.158904    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-web" containerID="cri-o://f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0" gracePeriod=600
Jul 16 21:50:14.159182 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.159016    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d" gracePeriod=600
Jul 16 21:50:14.159182 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.159025    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="config-reloader" containerID="cri-o://ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1" gracePeriod=600
Jul 16 21:50:14.159182 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.158900    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="thanos-sidecar" containerID="cri-o://1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc" gracePeriod=600
Jul 16 21:50:14.393643 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.393620    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:14.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410122    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-grpc-tls\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410163    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410193    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-kubelet-serving-ca-bundle\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410431 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410224    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-metrics-client-ca\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410431 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410248    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-2\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410431 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410274    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-1\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410431 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410296    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410553 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410422    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-serving-certs-ca-bundle\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410553 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410457    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-thanos-prometheus-http-client-file\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410553 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410499    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-trusted-ca-bundle\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410553 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410530    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-metrics-client-certs\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410668 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410560    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-tls-assets\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410668 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410601    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410668 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410631    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-kube-rbac-proxy\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410756 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410665    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bktqp\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-kube-api-access-bktqp\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410756 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410695    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-tls\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410756 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410722    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-web-config\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.410756 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410751    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-0\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.411042 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410774    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config-out\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.411042 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.410798    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-db\") pod \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\" (UID: \"29c221a9-b9b5-4d2d-81b3-59cc1b044238\") "
Jul 16 21:50:14.411150 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.411118    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:14.411150 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.411086    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:14.411249 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.411151    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:14.411402 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.411371    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:14.411491 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.411451    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:14.411548 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.411518    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:14.412625 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.412364    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:50:14.413260 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.413233    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.413520 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.413490    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:14.413520 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.413507    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config" (OuterVolumeSpecName: "config") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.415518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.414048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.415518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.414128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.415518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.414451    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.415518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.414745    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.415518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.415046    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.415518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.415385    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.416027 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.416005    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:50:14.416135 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.416046    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-kube-api-access-bktqp" (OuterVolumeSpecName: "kube-api-access-bktqp") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "kube-api-access-bktqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:50:14.416135 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.416112    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config-out" (OuterVolumeSpecName: "config-out") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:50:14.424832 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.424807    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-web-config" (OuterVolumeSpecName: "web-config") pod "29c221a9-b9b5-4d2d-81b3-59cc1b044238" (UID: "29c221a9-b9b5-4d2d-81b3-59cc1b044238"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:14.512171 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512142    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512171 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512167    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512171 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512177    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-metrics-client-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512186    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512196    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512204    2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512212    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512221    2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512229    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512239    2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-metrics-client-certs\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512248    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-tls-assets\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512257    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512266    2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-kube-rbac-proxy\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512274    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bktqp\" (UniqueName: \"kubernetes.io/projected/29c221a9-b9b5-4d2d-81b3-59cc1b044238-kube-api-access-bktqp\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512282    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512291    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-web-config\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512299    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512306    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-config-out\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512314    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29c221a9-b9b5-4d2d-81b3-59cc1b044238-prometheus-k8s-db\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:14.512349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:14.512322    2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29c221a9-b9b5-4d2d-81b3-59cc1b044238-secret-grpc-tls\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 21:50:15.243361 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243332    2555 generic.go:356] "Generic (PLEG): container finished" podID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d" exitCode=0
Jul 16 21:50:15.243361 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243356    2555 generic.go:356] "Generic (PLEG): container finished" podID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e" exitCode=0
Jul 16 21:50:15.243361 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243365    2555 generic.go:356] "Generic (PLEG): container finished" podID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0" exitCode=0
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243372    2555 generic.go:356] "Generic (PLEG): container finished" podID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc" exitCode=0
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243381    2555 generic.go:356] "Generic (PLEG): container finished" podID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1" exitCode=0
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243388    2555 generic.go:356] "Generic (PLEG): container finished" podID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84" exitCode=0
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243409    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"}
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243485    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"}
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243515    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"}
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"}
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243561    2555 scope.go:117] "RemoveContainer" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"}
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243660    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"}
Jul 16 21:50:15.243874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.243683    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29c221a9-b9b5-4d2d-81b3-59cc1b044238","Type":"ContainerDied","Data":"18c1618cd122a5fdf9086b7244a5bac3044f2bf7c20b4bf2c33d5100194d636c"}
Jul 16 21:50:15.251385 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.251366    2555 scope.go:117] "RemoveContainer" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.258133 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.258118    2555 scope.go:117] "RemoveContainer" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.264424 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.264403    2555 scope.go:117] "RemoveContainer" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.270951 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.270930    2555 scope.go:117] "RemoveContainer" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.277472 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.277456    2555 scope.go:117] "RemoveContainer" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.282293 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.282265    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:50:15.285323 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.285301    2555 scope.go:117] "RemoveContainer" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.287025 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.287008    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:50:15.292029 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.292013    2555 scope.go:117] "RemoveContainer" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.292292 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:50:15.292265    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": container with ID starting with fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d not found: ID does not exist" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.292345 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.292301    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"} err="failed to get container status \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": rpc error: code = NotFound desc = could not find container \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": container with ID starting with fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d not found: ID does not exist"
Jul 16 21:50:15.292345 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.292337    2555 scope.go:117] "RemoveContainer" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.292555 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:50:15.292540    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": container with ID starting with cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e not found: ID does not exist" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.292599 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.292562    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"} err="failed to get container status \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": rpc error: code = NotFound desc = could not find container \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": container with ID starting with cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e not found: ID does not exist"
Jul 16 21:50:15.292599 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.292579    2555 scope.go:117] "RemoveContainer" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.292802 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:50:15.292785    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": container with ID starting with f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0 not found: ID does not exist" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.292869 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.292810    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"} err="failed to get container status \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": rpc error: code = NotFound desc = could not find container \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": container with ID starting with f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0 not found: ID does not exist"
Jul 16 21:50:15.292869 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.292831    2555 scope.go:117] "RemoveContainer" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.293112 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:50:15.293095    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": container with ID starting with 1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc not found: ID does not exist" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.293158 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293115    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"} err="failed to get container status \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": rpc error: code = NotFound desc = could not find container \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": container with ID starting with 1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc not found: ID does not exist"
Jul 16 21:50:15.293158 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293130    2555 scope.go:117] "RemoveContainer" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.293351 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:50:15.293337    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": container with ID starting with ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1 not found: ID does not exist" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.293389 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293353    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"} err="failed to get container status \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": rpc error: code = NotFound desc = could not find container \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": container with ID starting with ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1 not found: ID does not exist"
Jul 16 21:50:15.293389 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293364    2555 scope.go:117] "RemoveContainer" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.293606 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:50:15.293588    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": container with ID starting with 27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84 not found: ID does not exist" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.293682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293611    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"} err="failed to get container status \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": rpc error: code = NotFound desc = could not find container \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": container with ID starting with 27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84 not found: ID does not exist"
Jul 16 21:50:15.293682 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293625    2555 scope.go:117] "RemoveContainer" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.293909 ip-10-0-143-124 kubenswrapper[2555]: E0716 21:50:15.293891    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": container with ID starting with ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a not found: ID does not exist" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.294021 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293918    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"} err="failed to get container status \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": rpc error: code = NotFound desc = could not find container \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": container with ID starting with ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a not found: ID does not exist"
Jul 16 21:50:15.294021 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.293933    2555 scope.go:117] "RemoveContainer" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.294199 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294182    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"} err="failed to get container status \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": rpc error: code = NotFound desc = could not find container \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": container with ID starting with fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d not found: ID does not exist"
Jul 16 21:50:15.294249 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294199    2555 scope.go:117] "RemoveContainer" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.294407 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294392    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"} err="failed to get container status \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": rpc error: code = NotFound desc = could not find container \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": container with ID starting with cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e not found: ID does not exist"
Jul 16 21:50:15.294448 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294407    2555 scope.go:117] "RemoveContainer" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.294629 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294608    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"} err="failed to get container status \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": rpc error: code = NotFound desc = could not find container \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": container with ID starting with f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0 not found: ID does not exist"
Jul 16 21:50:15.294629 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294627    2555 scope.go:117] "RemoveContainer" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.294852 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294834    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"} err="failed to get container status \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": rpc error: code = NotFound desc = could not find container \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": container with ID starting with 1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc not found: ID does not exist"
Jul 16 21:50:15.294893 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.294853    2555 scope.go:117] "RemoveContainer" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.295094 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295068    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"} err="failed to get container status \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": rpc error: code = NotFound desc = could not find container \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": container with ID starting with ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1 not found: ID does not exist"
Jul 16 21:50:15.295160 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295096    2555 scope.go:117] "RemoveContainer" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.295294 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295275    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"} err="failed to get container status \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": rpc error: code = NotFound desc = could not find container \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": container with ID starting with 27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84 not found: ID does not exist"
Jul 16 21:50:15.295331 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295298    2555 scope.go:117] "RemoveContainer" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.295515 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295498    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"} err="failed to get container status \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": rpc error: code = NotFound desc = could not find container \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": container with ID starting with ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a not found: ID does not exist"
Jul 16 21:50:15.295552 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295515    2555 scope.go:117] "RemoveContainer" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.295714 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295694    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"} err="failed to get container status \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": rpc error: code = NotFound desc = could not find container \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": container with ID starting with fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d not found: ID does not exist"
Jul 16 21:50:15.295754 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295715    2555 scope.go:117] "RemoveContainer" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.295980 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295947    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"} err="failed to get container status \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": rpc error: code = NotFound desc = could not find container \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": container with ID starting with cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e not found: ID does not exist"
Jul 16 21:50:15.296058 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.295982    2555 scope.go:117] "RemoveContainer" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.296229 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296213    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"} err="failed to get container status \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": rpc error: code = NotFound desc = could not find container \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": container with ID starting with f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0 not found: ID does not exist"
Jul 16 21:50:15.296272 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296229    2555 scope.go:117] "RemoveContainer" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.296429 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296412    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"} err="failed to get container status \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": rpc error: code = NotFound desc = could not find container \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": container with ID starting with 1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc not found: ID does not exist"
Jul 16 21:50:15.296478 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296430    2555 scope.go:117] "RemoveContainer" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.296602 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296586    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"} err="failed to get container status \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": rpc error: code = NotFound desc = could not find container \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": container with ID starting with ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1 not found: ID does not exist"
Jul 16 21:50:15.296650 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296602    2555 scope.go:117] "RemoveContainer" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.296805 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296790    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"} err="failed to get container status \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": rpc error: code = NotFound desc = could not find container \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": container with ID starting with 27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84 not found: ID does not exist"
Jul 16 21:50:15.296843 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.296805    2555 scope.go:117] "RemoveContainer" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.297024 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297007    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"} err="failed to get container status \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": rpc error: code = NotFound desc = could not find container \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": container with ID starting with ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a not found: ID does not exist"
Jul 16 21:50:15.297071 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297024    2555 scope.go:117] "RemoveContainer" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.297222 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297205    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"} err="failed to get container status \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": rpc error: code = NotFound desc = could not find container \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": container with ID starting with fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d not found: ID does not exist"
Jul 16 21:50:15.297257 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297223    2555 scope.go:117] "RemoveContainer" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.297434 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297416    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"} err="failed to get container status \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": rpc error: code = NotFound desc = could not find container \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": container with ID starting with cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e not found: ID does not exist"
Jul 16 21:50:15.297434 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297432    2555 scope.go:117] "RemoveContainer" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.297746 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297720    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"} err="failed to get container status \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": rpc error: code = NotFound desc = could not find container \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": container with ID starting with f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0 not found: ID does not exist"
Jul 16 21:50:15.297848 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.297746    2555 scope.go:117] "RemoveContainer" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.298059 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298040    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"} err="failed to get container status \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": rpc error: code = NotFound desc = could not find container \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": container with ID starting with 1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc not found: ID does not exist"
Jul 16 21:50:15.298134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298060    2555 scope.go:117] "RemoveContainer" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.298334 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298312    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"} err="failed to get container status \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": rpc error: code = NotFound desc = could not find container \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": container with ID starting with ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1 not found: ID does not exist"
Jul 16 21:50:15.298442 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298336    2555 scope.go:117] "RemoveContainer" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.298583 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298563    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"} err="failed to get container status \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": rpc error: code = NotFound desc = could not find container \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": container with ID starting with 27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84 not found: ID does not exist"
Jul 16 21:50:15.298583 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298583    2555 scope.go:117] "RemoveContainer" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.298861 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298839    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"} err="failed to get container status \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": rpc error: code = NotFound desc = could not find container \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": container with ID starting with ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a not found: ID does not exist"
Jul 16 21:50:15.298861 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.298860    2555 scope.go:117] "RemoveContainer" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.299136 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299107    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"} err="failed to get container status \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": rpc error: code = NotFound desc = could not find container \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": container with ID starting with fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d not found: ID does not exist"
Jul 16 21:50:15.299194 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299139    2555 scope.go:117] "RemoveContainer" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.299427 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299405    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"} err="failed to get container status \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": rpc error: code = NotFound desc = could not find container \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": container with ID starting with cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e not found: ID does not exist"
Jul 16 21:50:15.299491 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299429    2555 scope.go:117] "RemoveContainer" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.299663 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299643    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"} err="failed to get container status \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": rpc error: code = NotFound desc = could not find container \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": container with ID starting with f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0 not found: ID does not exist"
Jul 16 21:50:15.299663 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299662    2555 scope.go:117] "RemoveContainer" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.299867 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299847    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"} err="failed to get container status \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": rpc error: code = NotFound desc = could not find container \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": container with ID starting with 1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc not found: ID does not exist"
Jul 16 21:50:15.299923 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.299868    2555 scope.go:117] "RemoveContainer" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.300125 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300108    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"} err="failed to get container status \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": rpc error: code = NotFound desc = could not find container \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": container with ID starting with ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1 not found: ID does not exist"
Jul 16 21:50:15.300177 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300126    2555 scope.go:117] "RemoveContainer" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.300386 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300359    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"} err="failed to get container status \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": rpc error: code = NotFound desc = could not find container \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": container with ID starting with 27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84 not found: ID does not exist"
Jul 16 21:50:15.300457 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300387    2555 scope.go:117] "RemoveContainer" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.300632 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300615    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"} err="failed to get container status \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": rpc error: code = NotFound desc = could not find container \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": container with ID starting with ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a not found: ID does not exist"
Jul 16 21:50:15.300686 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300632    2555 scope.go:117] "RemoveContainer" containerID="fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"
Jul 16 21:50:15.300825 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300811    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d"} err="failed to get container status \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": rpc error: code = NotFound desc = could not find container \"fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d\": container with ID starting with fd9c9ff813a31b792dab982436fe8d507986e9e63b601d832246d089572b1d1d not found: ID does not exist"
Jul 16 21:50:15.300874 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.300825    2555 scope.go:117] "RemoveContainer" containerID="cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"
Jul 16 21:50:15.301038 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301022    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e"} err="failed to get container status \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": rpc error: code = NotFound desc = could not find container \"cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e\": container with ID starting with cb01a4a0ed13e4872110a3e796069c0983709a003de4de2fb19eca91f385eb1e not found: ID does not exist"
Jul 16 21:50:15.301098 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301039    2555 scope.go:117] "RemoveContainer" containerID="f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"
Jul 16 21:50:15.301254 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301233    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0"} err="failed to get container status \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": rpc error: code = NotFound desc = could not find container \"f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0\": container with ID starting with f9b18a49af4d4ccd2f74cc42b3c213ac0cdc80ea7404eb6905c8d00bd97e99e0 not found: ID does not exist"
Jul 16 21:50:15.301307 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301256    2555 scope.go:117] "RemoveContainer" containerID="1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"
Jul 16 21:50:15.301534 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301515    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc"} err="failed to get container status \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": rpc error: code = NotFound desc = could not find container \"1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc\": container with ID starting with 1a4f13bad88d6013a9f6093e74b2942a012a4641a1c5110f1df68e6b3e35d7fc not found: ID does not exist"
Jul 16 21:50:15.301611 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301535    2555 scope.go:117] "RemoveContainer" containerID="ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"
Jul 16 21:50:15.301797 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301772    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1"} err="failed to get container status \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": rpc error: code = NotFound desc = could not find container \"ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1\": container with ID starting with ce8cb948dfd1e8214ef8221a7fad7488815a3bed9060c96a42ff42c341cc22c1 not found: ID does not exist"
Jul 16 21:50:15.301797 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301786    2555 scope.go:117] "RemoveContainer" containerID="27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"
Jul 16 21:50:15.301995 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301979    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84"} err="failed to get container status \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": rpc error: code = NotFound desc = could not find container \"27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84\": container with ID starting with 27c649b0649d3bb59a8885c78a4ebdee880af225462d75b367b283762442fe84 not found: ID does not exist"
Jul 16 21:50:15.302044 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.301995    2555 scope.go:117] "RemoveContainer" containerID="ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"
Jul 16 21:50:15.302218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.302193    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a"} err="failed to get container status \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": rpc error: code = NotFound desc = could not find container \"ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a\": container with ID starting with ef87fa15b29ef8932b56ec47bee1cd5a5d90e315901081ee4e3321c28f62089a not found: ID does not exist"
Jul 16 21:50:15.344382 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344345    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:50:15.344581 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344568    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="config-reloader"
Jul 16 21:50:15.344627 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344583    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="config-reloader"
Jul 16 21:50:15.344627 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344590    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="thanos-sidecar"
Jul 16 21:50:15.344627 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344596    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="thanos-sidecar"
Jul 16 21:50:15.344627 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344602    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy"
Jul 16 21:50:15.344627 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344607    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy"
Jul 16 21:50:15.344627 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344618    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d46f06-67bd-4895-ad89-91bec576c8dd" containerName="registry"
Jul 16 21:50:15.344627 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344624    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d46f06-67bd-4895-ad89-91bec576c8dd" containerName="registry"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344633    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="prometheus"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344640    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="prometheus"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344646    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-web"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344651    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-web"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344659    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="init-config-reloader"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344664    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="init-config-reloader"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344669    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-thanos"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344674    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-thanos"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344711    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="prometheus"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344718    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="config-reloader"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344723    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="thanos-sidecar"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344728    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-web"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344734    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d46f06-67bd-4895-ad89-91bec576c8dd" containerName="registry"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344740    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy"
Jul 16 21:50:15.344823 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.344745    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" containerName="kube-rbac-proxy-thanos"
Jul 16 21:50:15.349705 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.349686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.353098 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.353080    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 21:50:15.353220 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.353128    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 21:50:15.353371 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.353355    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 21:50:15.353460 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.353399    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 21:50:15.353518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.353507    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zksjq\""
Jul 16 21:50:15.353673 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.353660    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 21:50:15.353757 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.353708    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 21:50:15.354446 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.354383    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 21:50:15.356507 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.354803    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 21:50:15.356507 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.355216    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 21:50:15.356507 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.356075    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 21:50:15.356507 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.356796    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 21:50:15.356507 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.358029    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-42u5ha48sqvkh\""
Jul 16 21:50:15.358877 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.358832    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 21:50:15.360289 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.359804    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 21:50:15.361573 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.361553    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 21:50:15.362589 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.362575    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 21:50:15.374041 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.374017    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:50:15.420742 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420658    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac00706d-66c9-4892-a0a5-b7018dd478b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420742 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420701    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420742 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420746    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420788    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420822    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49j9f\" (UniqueName: \"kubernetes.io/projected/ac00706d-66c9-4892-a0a5-b7018dd478b1-kube-api-access-49j9f\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420896    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac00706d-66c9-4892-a0a5-b7018dd478b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.420950 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.420995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421142    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421180 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421382 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421196    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-config\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421382 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421217    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.421382 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.421230    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.521858 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.521779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.521858 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.521828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.521858 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.521847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-config\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.521858 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.521862    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522123 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.521878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522123 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522021    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac00706d-66c9-4892-a0a5-b7018dd478b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522123 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522071    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522123 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49j9f\" (UniqueName: \"kubernetes.io/projected/ac00706d-66c9-4892-a0a5-b7018dd478b1-kube-api-access-49j9f\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522253    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac00706d-66c9-4892-a0a5-b7018dd478b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522312 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522702 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522343    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522702 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522702 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522702 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522702 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522947 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.522947 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.522901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.523079 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.523054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.523483 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.523458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.523756 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.523731    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.525131 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.525105    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.525218 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.525153    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.525437 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.525414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.525595 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.525572    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-config\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.526279 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.525904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.526279 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.526171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.526721 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.526694    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac00706d-66c9-4892-a0a5-b7018dd478b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.526847 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.526826    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.526911 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.526835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac00706d-66c9-4892-a0a5-b7018dd478b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.527303 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.527285    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac00706d-66c9-4892-a0a5-b7018dd478b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.527924 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.527903    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.528102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.528087    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.528378 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.528362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac00706d-66c9-4892-a0a5-b7018dd478b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.535485 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.535462    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49j9f\" (UniqueName: \"kubernetes.io/projected/ac00706d-66c9-4892-a0a5-b7018dd478b1-kube-api-access-49j9f\") pod \"prometheus-k8s-0\" (UID: \"ac00706d-66c9-4892-a0a5-b7018dd478b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.662195 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.662145    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:50:15.799835 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:15.799809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 21:50:15.801231 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:50:15.801209    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac00706d_66c9_4892_a0a5_b7018dd478b1.slice/crio-06eda7f509a4521a093a14267f4ced9ecacac5f6be73c515f36884a78497d81d WatchSource:0}: Error finding container 06eda7f509a4521a093a14267f4ced9ecacac5f6be73c515f36884a78497d81d: Status 404 returned error can't find the container with id 06eda7f509a4521a093a14267f4ced9ecacac5f6be73c515f36884a78497d81d
Jul 16 21:50:16.249007 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:16.248974    2555 generic.go:356] "Generic (PLEG): container finished" podID="ac00706d-66c9-4892-a0a5-b7018dd478b1" containerID="77f607ad910f4bc89ca2ca6ef4185d6e85f640b45ad8d9bf505f651f79bff292" exitCode=0
Jul 16 21:50:16.249368 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:16.249013    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerDied","Data":"77f607ad910f4bc89ca2ca6ef4185d6e85f640b45ad8d9bf505f651f79bff292"}
Jul 16 21:50:16.249368 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:16.249063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerStarted","Data":"06eda7f509a4521a093a14267f4ced9ecacac5f6be73c515f36884a78497d81d"}
Jul 16 21:50:16.495931 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:16.495896    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c221a9-b9b5-4d2d-81b3-59cc1b044238" path="/var/lib/kubelet/pods/29c221a9-b9b5-4d2d-81b3-59cc1b044238/volumes"
Jul 16 21:50:17.255871 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:17.255835    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerStarted","Data":"dd32627b892f44adc72f8904d829759e879f6dec0fb7ba0d30a1b8217e1ef1dc"}
Jul 16 21:50:17.256287 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:17.255925    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerStarted","Data":"15d02a692f8b3c8685bcffd1a5ef5ce77346a01f543929b733c3fb14f6915ed9"}
Jul 16 21:50:17.256287 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:17.255977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerStarted","Data":"9d94ca79434b5376181757627a77c796e8a9bbd0afaf593c437c8c79f80b7244"}
Jul 16 21:50:17.256287 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:17.256004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerStarted","Data":"aad921263df339ed3522bb73ebd986d406e78de6d0ea902f46a498e0f18dee90"}
Jul 16 21:50:17.256287 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:17.256027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerStarted","Data":"f35b33b7306993c76485bda5a4cc9514e04599c37ad54270c49adf89589fe52e"}
Jul 16 21:50:17.256287 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:17.256050    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac00706d-66c9-4892-a0a5-b7018dd478b1","Type":"ContainerStarted","Data":"d8b6287fdf56cfaed06822a67bcd0a2068b640695c078a7c54782185be87b3a8"}
Jul 16 21:50:17.291892 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:17.291836    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.272817537 podStartE2EDuration="2.29181412s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.996583ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:17.288714578 +0000 UTC m=+173.281878752" watchObservedRunningTime="2026-07-16 21:50:17.29181412 +0000 UTC m=+173.284978299"
Jul 16 21:50:20.663235 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:50:20.663192    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:51:15.662396 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:51:15.662339    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:51:15.677209 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:51:15.677186    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:51:16.476037 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:51:16.476013    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 21:52:24.368308 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:52:24.368282    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:53:34.495463 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.495389    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"]
Jul 16 21:53:34.498538 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.498520    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.514223 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.514201    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 21:53:34.514343 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.514238    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 21:53:34.514343 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.514247    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 21:53:34.514343 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.514339    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9vl5d\""
Jul 16 21:53:34.525713 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.525693    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 21:53:34.538107 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.538085    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"]
Jul 16 21:53:34.573853 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.573830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/3c33481c-6a26-4613-98f7-df5125cf3b05-kube-api-access-d6tsm\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.574005 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.573893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c33481c-6a26-4613-98f7-df5125cf3b05-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.574077 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.574001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c33481c-6a26-4613-98f7-df5125cf3b05-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.574134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.574091    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c33481c-6a26-4613-98f7-df5125cf3b05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.574134 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.574124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c33481c-6a26-4613-98f7-df5125cf3b05-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.675166 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.675119    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c33481c-6a26-4613-98f7-df5125cf3b05-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.675166 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.675178    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c33481c-6a26-4613-98f7-df5125cf3b05-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.675430 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.675217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c33481c-6a26-4613-98f7-df5125cf3b05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.675430 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.675238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c33481c-6a26-4613-98f7-df5125cf3b05-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.675430 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.675261    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/3c33481c-6a26-4613-98f7-df5125cf3b05-kube-api-access-d6tsm\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.675591 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.675510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c33481c-6a26-4613-98f7-df5125cf3b05-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.677618 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.677585    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c33481c-6a26-4613-98f7-df5125cf3b05-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.677760 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.677676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c33481c-6a26-4613-98f7-df5125cf3b05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.677821 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.677787    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c33481c-6a26-4613-98f7-df5125cf3b05-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.692120 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.692095    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/3c33481c-6a26-4613-98f7-df5125cf3b05-kube-api-access-d6tsm\") pod \"opendatahub-operator-controller-manager-574666dc8f-77cfc\" (UID: \"3c33481c-6a26-4613-98f7-df5125cf3b05\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.809196 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.809111    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:34.945846 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.945822    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"]
Jul 16 21:53:34.948194 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:53:34.948167    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c33481c_6a26_4613_98f7_df5125cf3b05.slice/crio-418c14afe0e9baf4463911897a2b504f94264e9e5f3ce5278a2711c32dd30f2d WatchSource:0}: Error finding container 418c14afe0e9baf4463911897a2b504f94264e9e5f3ce5278a2711c32dd30f2d: Status 404 returned error can't find the container with id 418c14afe0e9baf4463911897a2b504f94264e9e5f3ce5278a2711c32dd30f2d
Jul 16 21:53:34.950154 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:34.950136    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:53:35.920004 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:35.919941    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc" event={"ID":"3c33481c-6a26-4613-98f7-df5125cf3b05","Type":"ContainerStarted","Data":"418c14afe0e9baf4463911897a2b504f94264e9e5f3ce5278a2711c32dd30f2d"}
Jul 16 21:53:37.928485 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:37.928452    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc" event={"ID":"3c33481c-6a26-4613-98f7-df5125cf3b05","Type":"ContainerStarted","Data":"12af9f38fa25e084ad07fd4f333e0e5b4b716a771a67323fce18102feb97976b"}
Jul 16 21:53:38.933102 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:38.933067    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c33481c-6a26-4613-98f7-df5125cf3b05" containerID="12af9f38fa25e084ad07fd4f333e0e5b4b716a771a67323fce18102feb97976b" exitCode=0
Jul 16 21:53:38.933456 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:38.933137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc" event={"ID":"3c33481c-6a26-4613-98f7-df5125cf3b05","Type":"ContainerDied","Data":"12af9f38fa25e084ad07fd4f333e0e5b4b716a771a67323fce18102feb97976b"}
Jul 16 21:53:39.939059 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:39.939023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc" event={"ID":"3c33481c-6a26-4613-98f7-df5125cf3b05","Type":"ContainerStarted","Data":"4e6b86727ff4e3feed398c2427af81ff85b33e0874b6802f20c04bb089291f8e"}
Jul 16 21:53:39.939488 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:39.939164    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:53:40.041242 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:40.041178    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc" podStartSLOduration=1.8430471179999999 podStartE2EDuration="6.041153701s" totalImagesPullingTime="3.18376708s" totalInitContainerRuntime="1.014339503s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:40.037195486 +0000 UTC m=+376.030359640" watchObservedRunningTime="2026-07-16 21:53:40.041153701 +0000 UTC m=+376.034317856"
Jul 16 21:53:50.945519 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:53:50.945486    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-77cfc"
Jul 16 21:54:02.774622 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.774588    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"]
Jul 16 21:54:02.782921 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.782900    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:02.789069 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.789044    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 21:54:02.789175 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.789141    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 21:54:02.789778 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.789753    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 21:54:02.789778 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.789755    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qq7v5\""
Jul 16 21:54:02.789926 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.789809    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 21:54:02.798695 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.798675    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:54:02.803179 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.803161    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"]
Jul 16 21:54:02.911421 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.911383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd74f09b-c195-4619-b8dd-f2a00231df4b-metrics-cert\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:02.911597 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.911447    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd74f09b-c195-4619-b8dd-f2a00231df4b-cert\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:02.911597 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.911476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftbt\" (UniqueName: \"kubernetes.io/projected/fd74f09b-c195-4619-b8dd-f2a00231df4b-kube-api-access-rftbt\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:02.911597 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:02.911523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd74f09b-c195-4619-b8dd-f2a00231df4b-manager-config\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.012347 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.012308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd74f09b-c195-4619-b8dd-f2a00231df4b-cert\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.012347 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.012347    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftbt\" (UniqueName: \"kubernetes.io/projected/fd74f09b-c195-4619-b8dd-f2a00231df4b-kube-api-access-rftbt\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.012557 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.012405    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd74f09b-c195-4619-b8dd-f2a00231df4b-manager-config\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.012557 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.012437    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd74f09b-c195-4619-b8dd-f2a00231df4b-metrics-cert\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.013181 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.013160    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd74f09b-c195-4619-b8dd-f2a00231df4b-manager-config\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.015076 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.015047    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd74f09b-c195-4619-b8dd-f2a00231df4b-metrics-cert\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.015190 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.015046    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd74f09b-c195-4619-b8dd-f2a00231df4b-cert\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.040329 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.040254    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftbt\" (UniqueName: \"kubernetes.io/projected/fd74f09b-c195-4619-b8dd-f2a00231df4b-kube-api-access-rftbt\") pod \"lws-controller-manager-6bfbbdbb89-tdkgr\" (UID: \"fd74f09b-c195-4619-b8dd-f2a00231df4b\") " pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.091929 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.091897    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:03.244258 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:54:03.244229    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd74f09b_c195_4619_b8dd_f2a00231df4b.slice/crio-0d0012fe400a307f56cc35590c8ee192a7fb36fbbfaea05f03b2f56b0aff1011 WatchSource:0}: Error finding container 0d0012fe400a307f56cc35590c8ee192a7fb36fbbfaea05f03b2f56b0aff1011: Status 404 returned error can't find the container with id 0d0012fe400a307f56cc35590c8ee192a7fb36fbbfaea05f03b2f56b0aff1011
Jul 16 21:54:03.254129 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.254104    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"]
Jul 16 21:54:03.352739 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.352706    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"]
Jul 16 21:54:03.356906 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.356885    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.362672 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.362650    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gn6s5\""
Jul 16 21:54:03.363009 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.362993    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 21:54:03.363713 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.363697    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 21:54:03.363797 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.363734    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 21:54:03.364925 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.364903    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 21:54:03.365744 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.365730    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 21:54:03.369495 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.369481    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 21:54:03.424168 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.424133    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"]
Jul 16 21:54:03.515518 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.515485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.515691 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.515524    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/e1494e72-6683-4c57-a548-70f175887e9f-kube-api-access-gpnjg\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.515691 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.515573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1494e72-6683-4c57-a548-70f175887e9f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.515691 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.515616    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.515691 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.515667    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1494e72-6683-4c57-a548-70f175887e9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.515876 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.515696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.515876 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.515753    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1494e72-6683-4c57-a548-70f175887e9f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.616349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.616257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1494e72-6683-4c57-a548-70f175887e9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.616349 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.616317    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.616590 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.616382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1494e72-6683-4c57-a548-70f175887e9f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.616590 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.616509    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.616590 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.616560    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/e1494e72-6683-4c57-a548-70f175887e9f-kube-api-access-gpnjg\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.616745 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.616612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1494e72-6683-4c57-a548-70f175887e9f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.616745 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.616691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.617081 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.617031    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1494e72-6683-4c57-a548-70f175887e9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.618893 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.618865    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.619029 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.619002    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.619128 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.619111    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1494e72-6683-4c57-a548-70f175887e9f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.619168 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.619154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1494e72-6683-4c57-a548-70f175887e9f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.630933 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.630912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1494e72-6683-4c57-a548-70f175887e9f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.632260 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.632244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/e1494e72-6683-4c57-a548-70f175887e9f-kube-api-access-gpnjg\") pod \"istiod-openshift-gateway-94bb8fbfd-xfvv7\" (UID: \"e1494e72-6683-4c57-a548-70f175887e9f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.666293 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.666261    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:03.839501 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:03.839421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"]
Jul 16 21:54:03.843216 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:54:03.843183    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1494e72_6683_4c57_a548_70f175887e9f.slice/crio-59d7f93f1712bf51188d1a815610965c61fa84837173bb4fcd66d756f08fd18c WatchSource:0}: Error finding container 59d7f93f1712bf51188d1a815610965c61fa84837173bb4fcd66d756f08fd18c: Status 404 returned error can't find the container with id 59d7f93f1712bf51188d1a815610965c61fa84837173bb4fcd66d756f08fd18c
Jul 16 21:54:04.026865 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:04.026738    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7" event={"ID":"e1494e72-6683-4c57-a548-70f175887e9f","Type":"ContainerStarted","Data":"59d7f93f1712bf51188d1a815610965c61fa84837173bb4fcd66d756f08fd18c"}
Jul 16 21:54:04.028522 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:04.028469    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr" event={"ID":"fd74f09b-c195-4619-b8dd-f2a00231df4b","Type":"ContainerStarted","Data":"0d0012fe400a307f56cc35590c8ee192a7fb36fbbfaea05f03b2f56b0aff1011"}
Jul 16 21:54:06.722070 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:06.722035    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 16 21:54:06.722362 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:06.722102    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 16 21:54:07.043428 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:07.043336    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7" event={"ID":"e1494e72-6683-4c57-a548-70f175887e9f","Type":"ContainerStarted","Data":"f0e7f0e1cbffd91cd3fd1442890e14e40b550b38111f9b33e80cbf2ecae7c7e3"}
Jul 16 21:54:07.043585 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:07.043489    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:07.045261 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:07.045235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr" event={"ID":"fd74f09b-c195-4619-b8dd-f2a00231df4b","Type":"ContainerStarted","Data":"43993b0243c97ca0d2e2a944ca654dbb7271487383f05e75d992b3c76e299781"}
Jul 16 21:54:07.045400 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:07.045344    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:54:07.045461 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:07.045394    2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-xfvv7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 21:54:07.045461 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:07.045438    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7" podUID="e1494e72-6683-4c57-a548-70f175887e9f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 21:54:07.207912 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:07.207856    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7" podStartSLOduration=1.332306613 podStartE2EDuration="4.207840815s" totalImagesPullingTime="2.875534202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:07.19978515 +0000 UTC m=+403.192949314" watchObservedRunningTime="2026-07-16 21:54:07.207840815 +0000 UTC m=+403.201004967"
Jul 16 21:54:08.049362 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:08.049332    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-xfvv7"
Jul 16 21:54:08.090704 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:08.090646    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr" podStartSLOduration=2.615939211 podStartE2EDuration="6.090631119s" totalImagesPullingTime="3.474691908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:07.340166528 +0000 UTC m=+403.333330680" watchObservedRunningTime="2026-07-16 21:54:08.090631119 +0000 UTC m=+404.083795271"
Jul 16 21:54:18.051466 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:54:18.051437    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bfbbdbb89-tdkgr"
Jul 16 21:55:54.406605 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.406573    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"]
Jul 16 21:55:54.409789 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.409773    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.413917 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.413892    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wqpb2\""
Jul 16 21:55:54.414045 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.413973    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:55:54.414522 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.414493    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:55:54.436063 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.436038    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"]
Jul 16 21:55:54.475357 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.475330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f767x\" (UniqueName: \"kubernetes.io/projected/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-kube-api-access-f767x\") pod \"kuadrant-operator-controller-manager-86878c769f-8h7j5\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.475507 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.475383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8h7j5\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.576184 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.576148    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f767x\" (UniqueName: \"kubernetes.io/projected/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-kube-api-access-f767x\") pod \"kuadrant-operator-controller-manager-86878c769f-8h7j5\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.576369 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.576204    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8h7j5\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.576591 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.576567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8h7j5\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.609482 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.609453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f767x\" (UniqueName: \"kubernetes.io/projected/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-kube-api-access-f767x\") pod \"kuadrant-operator-controller-manager-86878c769f-8h7j5\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.720222 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.720126    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:54.878293 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:54.878255    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"]
Jul 16 21:55:54.881540 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:55:54.881512    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a7d2f6_7190_4ca1_a5d9_894cbf97c83f.slice/crio-bd44c5beb00a814bac1786519d6c47ebaee7ad206d7db484793ed491fdce026f WatchSource:0}: Error finding container bd44c5beb00a814bac1786519d6c47ebaee7ad206d7db484793ed491fdce026f: Status 404 returned error can't find the container with id bd44c5beb00a814bac1786519d6c47ebaee7ad206d7db484793ed491fdce026f
Jul 16 21:55:55.432437 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:55.432399    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5" event={"ID":"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f","Type":"ContainerStarted","Data":"bd44c5beb00a814bac1786519d6c47ebaee7ad206d7db484793ed491fdce026f"}
Jul 16 21:55:59.451829 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:59.451792    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5" event={"ID":"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f","Type":"ContainerStarted","Data":"955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b"}
Jul 16 21:55:59.452305 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:59.451993    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:55:59.501943 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:55:59.501848    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5" podStartSLOduration=1.152605356 podStartE2EDuration="5.501833492s" totalImagesPullingTime="4.349228136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:59.500984676 +0000 UTC m=+515.494148830" watchObservedRunningTime="2026-07-16 21:55:59.501833492 +0000 UTC m=+515.494997643"
Jul 16 21:56:10.457765 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:56:10.457726    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 21:58:17.092384 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.092350    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"]
Jul 16 21:58:17.095554 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.095537    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.100341 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.100321    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 21:58:17.100467 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.100321    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 16 21:58:17.100467 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.100322    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5gqbs\""
Jul 16 21:58:17.100467 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.100352    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 21:58:17.108680 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.108657    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"]
Jul 16 21:58:17.173079 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.173047    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.173241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.173087    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.173241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.173115    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.173241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.173140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.173241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.173160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.173373 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.173237    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2dt\" (UniqueName: \"kubernetes.io/projected/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-kube-api-access-dg2dt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.273850 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.273815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.273850 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.273854    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.274122 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.273888    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2dt\" (UniqueName: \"kubernetes.io/projected/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-kube-api-access-dg2dt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.274122 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.273913    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.274122 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.273949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.274122 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.274004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.274366 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.274332    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.274366 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.274356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.274509 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.274440    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.276208 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.276188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.276499 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.276481    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.283715 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.283696    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2dt\" (UniqueName: \"kubernetes.io/projected/c16bf3b4-7e21-44d6-bc4a-7f89f3acccab-kube-api-access-dg2dt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-76krz\" (UID: \"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.406783 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.406690    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:17.541284 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.541259    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"]
Jul 16 21:58:17.543623 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:58:17.543594    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16bf3b4_7e21_44d6_bc4a_7f89f3acccab.slice/crio-5f49e83238205ba6e8e014371d3170f21a59063b4dfb4ebfaa4078b252e42ee5 WatchSource:0}: Error finding container 5f49e83238205ba6e8e014371d3170f21a59063b4dfb4ebfaa4078b252e42ee5: Status 404 returned error can't find the container with id 5f49e83238205ba6e8e014371d3170f21a59063b4dfb4ebfaa4078b252e42ee5
Jul 16 21:58:17.970617 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:17.970569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz" event={"ID":"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab","Type":"ContainerStarted","Data":"5f49e83238205ba6e8e014371d3170f21a59063b4dfb4ebfaa4078b252e42ee5"}
Jul 16 21:58:18.977357 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:18.977318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz" event={"ID":"c16bf3b4-7e21-44d6-bc4a-7f89f3acccab","Type":"ContainerStarted","Data":"b601d413f980022d483884308e8804061e3df7f65b21d8e5c9a570f7b16bd6f8"}
Jul 16 21:58:18.995241 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:18.995187    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz" podStartSLOduration=0.982781423 podStartE2EDuration="1.995171143s" totalImagesPullingTime="1.01238972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:18.99379511 +0000 UTC m=+654.986959274" watchObservedRunningTime="2026-07-16 21:58:18.995171143 +0000 UTC m=+654.988335332"
Jul 16 21:58:27.407757 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:27.407716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:27.408316 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:27.407798    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:27.420291 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:27.420267    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:28.026792 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:28.026766    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-76krz"
Jul 16 21:58:49.177855 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.177822    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"]
Jul 16 21:58:49.188977 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.188939    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.190479 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.190441    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"]
Jul 16 21:58:49.191787 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.191764    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 21:58:49.338625 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.338590    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.338816 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.338642    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a96291-b301-4202-b95f-d3d9bf38a5ed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.338816 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.338714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.338816 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.338734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.338816 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.338764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.338968 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.338852    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbpj\" (UniqueName: \"kubernetes.io/projected/48a96291-b301-4202-b95f-d3d9bf38a5ed-kube-api-access-9hbpj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440278 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440191    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a96291-b301-4202-b95f-d3d9bf38a5ed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440278 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440253    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440481 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440529 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440569 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440540    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbpj\" (UniqueName: \"kubernetes.io/projected/48a96291-b301-4202-b95f-d3d9bf38a5ed-kube-api-access-9hbpj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440617 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440591    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440671 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440631    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440742 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440719    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.440821 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.440799    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.442679 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.442656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a96291-b301-4202-b95f-d3d9bf38a5ed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.443138 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.443121    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a96291-b301-4202-b95f-d3d9bf38a5ed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.449007 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.448952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbpj\" (UniqueName: \"kubernetes.io/projected/48a96291-b301-4202-b95f-d3d9bf38a5ed-kube-api-access-9hbpj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24\" (UID: \"48a96291-b301-4202-b95f-d3d9bf38a5ed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.501015 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.500949    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:49.630760 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.630729    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"]
Jul 16 21:58:49.633096 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:58:49.633071    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a96291_b301_4202_b95f_d3d9bf38a5ed.slice/crio-66ff8dcb82d76ed03f8ed7a57ecc912daf7462df13ead59b6d6a48099c3947a9 WatchSource:0}: Error finding container 66ff8dcb82d76ed03f8ed7a57ecc912daf7462df13ead59b6d6a48099c3947a9: Status 404 returned error can't find the container with id 66ff8dcb82d76ed03f8ed7a57ecc912daf7462df13ead59b6d6a48099c3947a9
Jul 16 21:58:49.634462 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:49.634448    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:58:50.103453 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:50.103365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24" event={"ID":"48a96291-b301-4202-b95f-d3d9bf38a5ed","Type":"ContainerStarted","Data":"660cae356ddb3b7ad850f97d515c5e4c19ac820a9cfd9116a298266fe58a5c46"}
Jul 16 21:58:50.103453 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:50.103422    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24" event={"ID":"48a96291-b301-4202-b95f-d3d9bf38a5ed","Type":"ContainerStarted","Data":"66ff8dcb82d76ed03f8ed7a57ecc912daf7462df13ead59b6d6a48099c3947a9"}
Jul 16 21:58:50.122010 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:50.121948    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24" podStartSLOduration=0.916177229 podStartE2EDuration="1.121932461s" totalImagesPullingTime="205.755232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:50.120484783 +0000 UTC m=+686.113648935" watchObservedRunningTime="2026-07-16 21:58:50.121932461 +0000 UTC m=+686.115096613"
Jul 16 21:58:57.599549 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.599519    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"]
Jul 16 21:58:57.602839 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.602823    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.605994 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.605969    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 16 21:58:57.614717 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.614694    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"]
Jul 16 21:58:57.709785 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.709750    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc65c13-bf71-4849-875b-45293b86b4d3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.710009 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.709794    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.710009 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.709823    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.710009 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.709881    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6md\" (UniqueName: \"kubernetes.io/projected/8dc65c13-bf71-4849-875b-45293b86b4d3-kube-api-access-ln6md\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.710009 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.710005    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.710176 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.710035    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.810985 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.810927    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811168 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811168 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811039    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6md\" (UniqueName: \"kubernetes.io/projected/8dc65c13-bf71-4849-875b-45293b86b4d3-kube-api-access-ln6md\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811168 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811168 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811375 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811191    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc65c13-bf71-4849-875b-45293b86b4d3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811375 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811327    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811477 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.811535 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.811514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.813284 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.813264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc65c13-bf71-4849-875b-45293b86b4d3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.813561 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.813543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc65c13-bf71-4849-875b-45293b86b4d3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.818629 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.818603    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6md\" (UniqueName: \"kubernetes.io/projected/8dc65c13-bf71-4849-875b-45293b86b4d3-kube-api-access-ln6md\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tkmjx\" (UID: \"8dc65c13-bf71-4849-875b-45293b86b4d3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:57.914584 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:57.914503    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:58:58.046777 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:58.046692    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"]
Jul 16 21:58:58.049179 ip-10-0-143-124 kubenswrapper[2555]: W0716 21:58:58.049149    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc65c13_bf71_4849_875b_45293b86b4d3.slice/crio-908a18a9e5f946a56544e9ede5fceb341407d41d19ddf837fdae74e3307a7f6e WatchSource:0}: Error finding container 908a18a9e5f946a56544e9ede5fceb341407d41d19ddf837fdae74e3307a7f6e: Status 404 returned error can't find the container with id 908a18a9e5f946a56544e9ede5fceb341407d41d19ddf837fdae74e3307a7f6e
Jul 16 21:58:58.133100 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:58.133064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx" event={"ID":"8dc65c13-bf71-4849-875b-45293b86b4d3","Type":"ContainerStarted","Data":"908a18a9e5f946a56544e9ede5fceb341407d41d19ddf837fdae74e3307a7f6e"}
Jul 16 21:58:59.138761 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:59.138731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx" event={"ID":"8dc65c13-bf71-4849-875b-45293b86b4d3","Type":"ContainerStarted","Data":"2cb307e27a5dfc80409c4edef5fa9ab5dce3352c9242e391e0943ff710b7d942"}
Jul 16 21:58:59.159322 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:59.159271    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx" podStartSLOduration=1.959291887 podStartE2EDuration="2.159253005s" totalImagesPullingTime="199.961118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:59.157445809 +0000 UTC m=+695.150609962" watchObservedRunningTime="2026-07-16 21:58:59.159253005 +0000 UTC m=+695.152417157"
Jul 16 21:58:59.501592 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:59.501499    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:59.501749 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:59.501562    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:58:59.514345 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:58:59.514318    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:59:00.155761 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:59:00.155733    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24"
Jul 16 21:59:07.914794 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:59:07.914758    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:59:07.915243 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:59:07.914834    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:59:07.927226 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:59:07.927200    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 21:59:08.187111 ip-10-0-143-124 kubenswrapper[2555]: I0716 21:59:08.187029    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tkmjx"
Jul 16 22:07:14.667843 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:14.667808    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"]
Jul 16 22:07:14.668433 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:14.668016    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5" podUID="94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" containerName="manager" containerID="cri-o://955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b" gracePeriod=10
Jul 16 22:07:18.118504 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.118480    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 22:07:18.165416 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.165384    2555 generic.go:356] "Generic (PLEG): container finished" podID="94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" containerID="955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b" exitCode=0
Jul 16 22:07:18.165577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.165423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5" event={"ID":"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f","Type":"ContainerDied","Data":"955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b"}
Jul 16 22:07:18.165577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.165445    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"
Jul 16 22:07:18.165577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.165494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5" event={"ID":"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f","Type":"ContainerDied","Data":"bd44c5beb00a814bac1786519d6c47ebaee7ad206d7db484793ed491fdce026f"}
Jul 16 22:07:18.165577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.165517    2555 scope.go:117] "RemoveContainer" containerID="955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b"
Jul 16 22:07:18.174148 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.174129    2555 scope.go:117] "RemoveContainer" containerID="955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b"
Jul 16 22:07:18.174431 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:07:18.174402    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b\": container with ID starting with 955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b not found: ID does not exist" containerID="955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b"
Jul 16 22:07:18.174500 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.174442    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b"} err="failed to get container status \"955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b\": rpc error: code = NotFound desc = could not find container \"955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b\": container with ID starting with 955101c7ed0de0397c87722577f28b286ed8903dc299c21bae1dc511ba6e859b not found: ID does not exist"
Jul 16 22:07:18.199032 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.199002    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-extensions-socket-volume\") pod \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") "
Jul 16 22:07:18.199166 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.199056    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f767x\" (UniqueName: \"kubernetes.io/projected/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-kube-api-access-f767x\") pod \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\" (UID: \"94a7d2f6-7190-4ca1-a5d9-894cbf97c83f\") "
Jul 16 22:07:18.199431 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.199409    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" (UID: "94a7d2f6-7190-4ca1-a5d9-894cbf97c83f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:07:18.201170 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.201150    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-kube-api-access-f767x" (OuterVolumeSpecName: "kube-api-access-f767x") pod "94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" (UID: "94a7d2f6-7190-4ca1-a5d9-894cbf97c83f"). InnerVolumeSpecName "kube-api-access-f767x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:07:18.299628 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.299543    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f767x\" (UniqueName: \"kubernetes.io/projected/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-kube-api-access-f767x\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:07:18.299628 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.299576    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f-extensions-socket-volume\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:07:18.488456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.488424    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"]
Jul 16 22:07:18.494850 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:18.494821    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8h7j5"]
Jul 16 22:07:20.494235 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:07:20.494201    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" path="/var/lib/kubelet/pods/94a7d2f6-7190-4ca1-a5d9-894cbf97c83f/volumes"
Jul 16 22:08:15.851391 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.851300    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"]
Jul 16 22:08:15.851898 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.851754    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" containerName="manager"
Jul 16 22:08:15.851898 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.851774    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" containerName="manager"
Jul 16 22:08:15.851898 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.851860    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a7d2f6-7190-4ca1-a5d9-894cbf97c83f" containerName="manager"
Jul 16 22:08:15.854820 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.854800    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:15.857879 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.857858    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wqpb2\""
Jul 16 22:08:15.858233 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.858213    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 22:08:15.858621 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.858606    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 22:08:15.877705 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.877674    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"]
Jul 16 22:08:15.965365 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.965329    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nd44\" (UniqueName: \"kubernetes.io/projected/c598ac2d-2702-4336-bb1b-ce0fb66ed5a5-kube-api-access-4nd44\") pod \"kuadrant-operator-controller-manager-86878c769f-szbm8\" (UID: \"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:15.965532 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:15.965376    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c598ac2d-2702-4336-bb1b-ce0fb66ed5a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szbm8\" (UID: \"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:16.066039 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.066004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nd44\" (UniqueName: \"kubernetes.io/projected/c598ac2d-2702-4336-bb1b-ce0fb66ed5a5-kube-api-access-4nd44\") pod \"kuadrant-operator-controller-manager-86878c769f-szbm8\" (UID: \"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:16.066177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.066066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c598ac2d-2702-4336-bb1b-ce0fb66ed5a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szbm8\" (UID: \"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:16.066436 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.066417    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c598ac2d-2702-4336-bb1b-ce0fb66ed5a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-szbm8\" (UID: \"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:16.077441 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.077413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nd44\" (UniqueName: \"kubernetes.io/projected/c598ac2d-2702-4336-bb1b-ce0fb66ed5a5-kube-api-access-4nd44\") pod \"kuadrant-operator-controller-manager-86878c769f-szbm8\" (UID: \"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:16.164539 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.164451    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:16.301003 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.300981    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"]
Jul 16 22:08:16.303285 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:08:16.303244    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc598ac2d_2702_4336_bb1b_ce0fb66ed5a5.slice/crio-19ef32833a9c887c95a0dee49def98dc67d112160f66e3520668ada343518fc0 WatchSource:0}: Error finding container 19ef32833a9c887c95a0dee49def98dc67d112160f66e3520668ada343518fc0: Status 404 returned error can't find the container with id 19ef32833a9c887c95a0dee49def98dc67d112160f66e3520668ada343518fc0
Jul 16 22:08:16.379995 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.379934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8" event={"ID":"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5","Type":"ContainerStarted","Data":"d47b6783f2c286e33ad60d2759fc4c7b1bcfe2f874b106e256757abae2ebd45a"}
Jul 16 22:08:16.380178 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.380008    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8" event={"ID":"c598ac2d-2702-4336-bb1b-ce0fb66ed5a5","Type":"ContainerStarted","Data":"19ef32833a9c887c95a0dee49def98dc67d112160f66e3520668ada343518fc0"}
Jul 16 22:08:16.380178 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.380091    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:08:16.403248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:16.403196    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8" podStartSLOduration=1.403179376 podStartE2EDuration="1.403179376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:16.40167396 +0000 UTC m=+1252.394838115" watchObservedRunningTime="2026-07-16 22:08:16.403179376 +0000 UTC m=+1252.396343530"
Jul 16 22:08:27.386176 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:08:27.386140    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-szbm8"
Jul 16 22:15:00.139524 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:00.139485    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"]
Jul 16 22:15:00.142899 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:00.142874    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"
Jul 16 22:15:00.146403 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:00.146375    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:15:00.152526 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:00.152509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"
Jul 16 22:15:00.157804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:00.157778    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"]
Jul 16 22:15:00.319467 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:00.319444    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"]
Jul 16 22:15:00.322127 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:15:00.322097    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a07b8b_2dda_47bd_8c55_cd67d7a230c4.slice/crio-64216c5a95c83539b502525c71966e362a2b7c68a581a16ded3566b61ca39c36 WatchSource:0}: Error finding container 64216c5a95c83539b502525c71966e362a2b7c68a581a16ded3566b61ca39c36: Status 404 returned error can't find the container with id 64216c5a95c83539b502525c71966e362a2b7c68a581a16ded3566b61ca39c36
Jul 16 22:15:00.324091 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:00.324072    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:15:01.041509 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:01.041469    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" event={"ID":"e0a07b8b-2dda-47bd-8c55-cd67d7a230c4","Type":"ContainerStarted","Data":"64216c5a95c83539b502525c71966e362a2b7c68a581a16ded3566b61ca39c36"}
Jul 16 22:15:03.053168 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:03.053141    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerID="673ebc12cc0324aa464819eb7ce78bb82ad4210d771a3340fcc27179406bb250" exitCode=7
Jul 16 22:15:03.053537 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:03.053221    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" event={"ID":"e0a07b8b-2dda-47bd-8c55-cd67d7a230c4","Type":"ContainerDied","Data":"673ebc12cc0324aa464819eb7ce78bb82ad4210d771a3340fcc27179406bb250"}
Jul 16 22:15:03.053537 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:03.053381    2555 scope.go:117] "RemoveContainer" containerID="673ebc12cc0324aa464819eb7ce78bb82ad4210d771a3340fcc27179406bb250"
Jul 16 22:15:04.059665 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:04.059581    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerID="6122990625d86941241677b168c48869531767216dd4f6e9012f1bc928a2e31e" exitCode=7
Jul 16 22:15:04.060081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:04.059658    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" event={"ID":"e0a07b8b-2dda-47bd-8c55-cd67d7a230c4","Type":"ContainerDied","Data":"6122990625d86941241677b168c48869531767216dd4f6e9012f1bc928a2e31e"}
Jul 16 22:15:04.060081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:04.059712    2555 scope.go:117] "RemoveContainer" containerID="673ebc12cc0324aa464819eb7ce78bb82ad4210d771a3340fcc27179406bb250"
Jul 16 22:15:04.060081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:04.059914    2555 scope.go:117] "RemoveContainer" containerID="6122990625d86941241677b168c48869531767216dd4f6e9012f1bc928a2e31e"
Jul 16 22:15:04.060218 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:15:04.060192    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj_odh-ai-gateway-infra(e0a07b8b-2dda-47bd-8c55-cd67d7a230c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4"
Jul 16 22:15:05.065724 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:05.065698    2555 scope.go:117] "RemoveContainer" containerID="6122990625d86941241677b168c48869531767216dd4f6e9012f1bc928a2e31e"
Jul 16 22:15:05.066121 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:15:05.065894    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj_odh-ai-gateway-infra(e0a07b8b-2dda-47bd-8c55-cd67d7a230c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4"
Jul 16 22:15:20.489435 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:20.489400    2555 scope.go:117] "RemoveContainer" containerID="6122990625d86941241677b168c48869531767216dd4f6e9012f1bc928a2e31e"
Jul 16 22:15:21.131868 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:21.131836    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" event={"ID":"e0a07b8b-2dda-47bd-8c55-cd67d7a230c4","Type":"ContainerStarted","Data":"1a14fcb5af017a5993c3dce4d2a953a77de820c06ce20e9689f8845bbde169f6"}
Jul 16 22:15:21.149472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:21.149423    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" podStartSLOduration=19.092306873 podStartE2EDuration="21.149406506s" totalImagesPullingTime="2.057099633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:15:21.148214604 +0000 UTC m=+1677.141378759" watchObservedRunningTime="2026-07-16 22:15:21.149406506 +0000 UTC m=+1677.142570658"
Jul 16 22:15:22.137779 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:22.137747    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerID="1a14fcb5af017a5993c3dce4d2a953a77de820c06ce20e9689f8845bbde169f6" exitCode=7
Jul 16 22:15:22.138225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:22.137826    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" event={"ID":"e0a07b8b-2dda-47bd-8c55-cd67d7a230c4","Type":"ContainerDied","Data":"1a14fcb5af017a5993c3dce4d2a953a77de820c06ce20e9689f8845bbde169f6"}
Jul 16 22:15:22.138225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:22.137897    2555 scope.go:117] "RemoveContainer" containerID="6122990625d86941241677b168c48869531767216dd4f6e9012f1bc928a2e31e"
Jul 16 22:15:22.138225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:22.138150    2555 scope.go:117] "RemoveContainer" containerID="1a14fcb5af017a5993c3dce4d2a953a77de820c06ce20e9689f8845bbde169f6"
Jul 16 22:15:22.138390 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:15:22.138372    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj_odh-ai-gateway-infra(e0a07b8b-2dda-47bd-8c55-cd67d7a230c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4"
Jul 16 22:15:22.162914 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:22.162887    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"]
Jul 16 22:15:23.270249 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:23.270225    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"
Jul 16 22:15:24.148137 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:24.148107    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"
Jul 16 22:15:24.148137 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:24.148131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj" event={"ID":"e0a07b8b-2dda-47bd-8c55-cd67d7a230c4","Type":"ContainerDied","Data":"64216c5a95c83539b502525c71966e362a2b7c68a581a16ded3566b61ca39c36"}
Jul 16 22:15:24.148383 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:24.148187    2555 scope.go:117] "RemoveContainer" containerID="1a14fcb5af017a5993c3dce4d2a953a77de820c06ce20e9689f8845bbde169f6"
Jul 16 22:15:24.169005 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:24.168981    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"]
Jul 16 22:15:24.177676 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:24.177629    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-eea3cfe3-29737335-w6srj"]
Jul 16 22:15:24.494894 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:24.494816    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" path="/var/lib/kubelet/pods/e0a07b8b-2dda-47bd-8c55-cd67d7a230c4/volumes"
Jul 16 22:15:36.065247 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065212    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"]
Jul 16 22:15:36.065624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065498    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:15:36.065624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065508    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:15:36.065624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065532    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:15:36.065624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065538    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:15:36.065624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065585    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:15:36.065624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065593    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:15:36.065624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.065600    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:15:36.069049 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.069033    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"
Jul 16 22:15:36.071821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.071796    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:15:36.071867 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.071837    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:15:36.076608 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.076581    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"]
Jul 16 22:15:36.147981 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.147920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx\" (UID: \"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"
Jul 16 22:15:36.249067 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.249022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx\" (UID: \"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"
Jul 16 22:15:36.249708 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.249690    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx\" (UID: \"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"
Jul 16 22:15:36.380902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.380866    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"
Jul 16 22:15:36.515258 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:36.515234    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"]
Jul 16 22:15:36.517353 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:15:36.517323    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c64eea_b0ea_44de_8b8a_9b133e1ad9d2.slice/crio-26d3c55c18f42cc02b32306ab1a8e0f51a961d79c46bc2b905d71482e1241835 WatchSource:0}: Error finding container 26d3c55c18f42cc02b32306ab1a8e0f51a961d79c46bc2b905d71482e1241835: Status 404 returned error can't find the container with id 26d3c55c18f42cc02b32306ab1a8e0f51a961d79c46bc2b905d71482e1241835
Jul 16 22:15:37.202468 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:37.202435    2555 generic.go:356] "Generic (PLEG): container finished" podID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerID="50646a6014a2fbdc520cb3325c0b858f3447c0dec14ea6584be95b1ba1af4ba6" exitCode=7
Jul 16 22:15:37.202945 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:37.202524    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" event={"ID":"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2","Type":"ContainerDied","Data":"50646a6014a2fbdc520cb3325c0b858f3447c0dec14ea6584be95b1ba1af4ba6"}
Jul 16 22:15:37.202945 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:37.202606    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" event={"ID":"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2","Type":"ContainerStarted","Data":"26d3c55c18f42cc02b32306ab1a8e0f51a961d79c46bc2b905d71482e1241835"}
Jul 16 22:15:37.202945 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:37.202733    2555 scope.go:117] "RemoveContainer" containerID="50646a6014a2fbdc520cb3325c0b858f3447c0dec14ea6584be95b1ba1af4ba6"
Jul 16 22:15:38.208716 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:38.208685    2555 generic.go:356] "Generic (PLEG): container finished" podID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerID="df2d9a5e12b0e0f7b9d8a5027ded599cf7cd68152f06274b7ba5860279a8e4c9" exitCode=7
Jul 16 22:15:38.209208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:38.208763    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" event={"ID":"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2","Type":"ContainerDied","Data":"df2d9a5e12b0e0f7b9d8a5027ded599cf7cd68152f06274b7ba5860279a8e4c9"}
Jul 16 22:15:38.209208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:38.208824    2555 scope.go:117] "RemoveContainer" containerID="50646a6014a2fbdc520cb3325c0b858f3447c0dec14ea6584be95b1ba1af4ba6"
Jul 16 22:15:38.209208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:38.209077    2555 scope.go:117] "RemoveContainer" containerID="df2d9a5e12b0e0f7b9d8a5027ded599cf7cd68152f06274b7ba5860279a8e4c9"
Jul 16 22:15:38.209330 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:15:38.209280    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx_odh-ai-gateway-infra(f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2"
Jul 16 22:15:39.214335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:39.214310    2555 scope.go:117] "RemoveContainer" containerID="df2d9a5e12b0e0f7b9d8a5027ded599cf7cd68152f06274b7ba5860279a8e4c9"
Jul 16 22:15:39.214739 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:15:39.214491    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx_odh-ai-gateway-infra(f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2"
Jul 16 22:15:49.490464 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:49.490381    2555 scope.go:117] "RemoveContainer" containerID="df2d9a5e12b0e0f7b9d8a5027ded599cf7cd68152f06274b7ba5860279a8e4c9"
Jul 16 22:15:50.261429 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:50.261394    2555 generic.go:356] "Generic (PLEG): container finished" podID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerID="77e266b3f01ca216e617a8aaf396010b653f8c41754567b6d123c7f4b166d552" exitCode=7
Jul 16 22:15:50.261613 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:50.261433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" event={"ID":"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2","Type":"ContainerDied","Data":"77e266b3f01ca216e617a8aaf396010b653f8c41754567b6d123c7f4b166d552"}
Jul 16 22:15:50.261613 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:50.261481    2555 scope.go:117] "RemoveContainer" containerID="df2d9a5e12b0e0f7b9d8a5027ded599cf7cd68152f06274b7ba5860279a8e4c9"
Jul 16 22:15:50.261829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:50.261812    2555 scope.go:117] "RemoveContainer" containerID="77e266b3f01ca216e617a8aaf396010b653f8c41754567b6d123c7f4b166d552"
Jul 16 22:15:50.262073 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:15:50.262052    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx_odh-ai-gateway-infra(f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2"
Jul 16 22:15:51.289812 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:51.289775    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"]
Jul 16 22:15:51.423205 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:51.423181    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"
Jul 16 22:15:51.464705 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:51.464682    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2-maas-api-service-ca\") pod \"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2\" (UID: \"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2\") "
Jul 16 22:15:51.465065 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:51.465042    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" (UID: "f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:15:51.566201 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:51.566119    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:15:52.272638 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:52.272606    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"
Jul 16 22:15:52.272835 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:52.272637    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx" event={"ID":"f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2","Type":"ContainerDied","Data":"26d3c55c18f42cc02b32306ab1a8e0f51a961d79c46bc2b905d71482e1241835"}
Jul 16 22:15:52.272835 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:52.272704    2555 scope.go:117] "RemoveContainer" containerID="77e266b3f01ca216e617a8aaf396010b653f8c41754567b6d123c7f4b166d552"
Jul 16 22:15:52.294070 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:52.294042    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"]
Jul 16 22:15:52.300569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:52.300546    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpzdx"]
Jul 16 22:15:52.495057 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:15:52.495019    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" path="/var/lib/kubelet/pods/f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2/volumes"
Jul 16 22:16:26.106916 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.106879    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"]
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107184    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107196    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107214    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107220    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a07b8b-2dda-47bd-8c55-cd67d7a230c4" containerName="cleanup"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107229    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107234    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107281    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107289    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:16:26.107332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.107295    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:16:26.110102 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.110084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"
Jul 16 22:16:26.112668 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.112644    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:16:26.112794 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.112644    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:16:26.118174 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.118155    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"]
Jul 16 22:16:26.174303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.174266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78d645-1710-4ba4-a4ce-54dda0e888fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j\" (UID: \"ce78d645-1710-4ba4-a4ce-54dda0e888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"
Jul 16 22:16:26.274902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.274866    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78d645-1710-4ba4-a4ce-54dda0e888fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j\" (UID: \"ce78d645-1710-4ba4-a4ce-54dda0e888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"
Jul 16 22:16:26.275489 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.275470    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78d645-1710-4ba4-a4ce-54dda0e888fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j\" (UID: \"ce78d645-1710-4ba4-a4ce-54dda0e888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"
Jul 16 22:16:26.420984 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.420874    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"
Jul 16 22:16:26.549566 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:26.549507    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"]
Jul 16 22:16:26.551932 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:16:26.551905    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce78d645_1710_4ba4_a4ce_54dda0e888fa.slice/crio-7c4a990aff9e8fbd89714ec8092c1d4968842ad9ce332ea2766afb18ab3fee32 WatchSource:0}: Error finding container 7c4a990aff9e8fbd89714ec8092c1d4968842ad9ce332ea2766afb18ab3fee32: Status 404 returned error can't find the container with id 7c4a990aff9e8fbd89714ec8092c1d4968842ad9ce332ea2766afb18ab3fee32
Jul 16 22:16:27.412851 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:27.412758    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerID="e6c8b192d170f03d122b716d2bce07ead4e7bc81892b4719ba826016ad33a26a" exitCode=7
Jul 16 22:16:27.412851 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:27.412825    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" event={"ID":"ce78d645-1710-4ba4-a4ce-54dda0e888fa","Type":"ContainerDied","Data":"e6c8b192d170f03d122b716d2bce07ead4e7bc81892b4719ba826016ad33a26a"}
Jul 16 22:16:27.413291 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:27.412866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" event={"ID":"ce78d645-1710-4ba4-a4ce-54dda0e888fa","Type":"ContainerStarted","Data":"7c4a990aff9e8fbd89714ec8092c1d4968842ad9ce332ea2766afb18ab3fee32"}
Jul 16 22:16:27.413291 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:27.413064    2555 scope.go:117] "RemoveContainer" containerID="e6c8b192d170f03d122b716d2bce07ead4e7bc81892b4719ba826016ad33a26a"
Jul 16 22:16:28.419335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:28.419299    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerID="7f58dac88800ea551b7fc3193142a08d5e7605d9045b08e6d0cd813cec04fda3" exitCode=7
Jul 16 22:16:28.419760 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:28.419348    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" event={"ID":"ce78d645-1710-4ba4-a4ce-54dda0e888fa","Type":"ContainerDied","Data":"7f58dac88800ea551b7fc3193142a08d5e7605d9045b08e6d0cd813cec04fda3"}
Jul 16 22:16:28.419760 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:28.419412    2555 scope.go:117] "RemoveContainer" containerID="e6c8b192d170f03d122b716d2bce07ead4e7bc81892b4719ba826016ad33a26a"
Jul 16 22:16:28.419760 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:28.419562    2555 scope.go:117] "RemoveContainer" containerID="7f58dac88800ea551b7fc3193142a08d5e7605d9045b08e6d0cd813cec04fda3"
Jul 16 22:16:28.419929 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:16:28.419787    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j_odh-ai-gateway-infra(ce78d645-1710-4ba4-a4ce-54dda0e888fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa"
Jul 16 22:16:29.424983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:29.424938    2555 scope.go:117] "RemoveContainer" containerID="7f58dac88800ea551b7fc3193142a08d5e7605d9045b08e6d0cd813cec04fda3"
Jul 16 22:16:29.425365 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:16:29.425149    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j_odh-ai-gateway-infra(ce78d645-1710-4ba4-a4ce-54dda0e888fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa"
Jul 16 22:16:40.490056 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:40.490024    2555 scope.go:117] "RemoveContainer" containerID="7f58dac88800ea551b7fc3193142a08d5e7605d9045b08e6d0cd813cec04fda3"
Jul 16 22:16:41.476141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:41.476054    2555 generic.go:356] "Generic (PLEG): container finished" podID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerID="4ead1de9c89ed86402ba1c5d0db50ad04b6d7dba6f461c7c38169a8206b1cdeb" exitCode=7
Jul 16 22:16:41.476141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:41.476122    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" event={"ID":"ce78d645-1710-4ba4-a4ce-54dda0e888fa","Type":"ContainerDied","Data":"4ead1de9c89ed86402ba1c5d0db50ad04b6d7dba6f461c7c38169a8206b1cdeb"}
Jul 16 22:16:41.476319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:41.476193    2555 scope.go:117] "RemoveContainer" containerID="7f58dac88800ea551b7fc3193142a08d5e7605d9045b08e6d0cd813cec04fda3"
Jul 16 22:16:41.476535 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:41.476513    2555 scope.go:117] "RemoveContainer" containerID="4ead1de9c89ed86402ba1c5d0db50ad04b6d7dba6f461c7c38169a8206b1cdeb"
Jul 16 22:16:41.476810 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:16:41.476742    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j_odh-ai-gateway-infra(ce78d645-1710-4ba4-a4ce-54dda0e888fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa"
Jul 16 22:16:42.503313 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:42.503281    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"]
Jul 16 22:16:42.629968 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:42.629930    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"
Jul 16 22:16:42.695835 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:42.695798    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78d645-1710-4ba4-a4ce-54dda0e888fa-maas-api-service-ca\") pod \"ce78d645-1710-4ba4-a4ce-54dda0e888fa\" (UID: \"ce78d645-1710-4ba4-a4ce-54dda0e888fa\") "
Jul 16 22:16:42.696228 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:42.696203    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce78d645-1710-4ba4-a4ce-54dda0e888fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce78d645-1710-4ba4-a4ce-54dda0e888fa" (UID: "ce78d645-1710-4ba4-a4ce-54dda0e888fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:16:42.796581 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:42.796495    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce78d645-1710-4ba4-a4ce-54dda0e888fa-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:16:43.487178 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:43.487150    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"
Jul 16 22:16:43.487355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:43.487174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j" event={"ID":"ce78d645-1710-4ba4-a4ce-54dda0e888fa","Type":"ContainerDied","Data":"7c4a990aff9e8fbd89714ec8092c1d4968842ad9ce332ea2766afb18ab3fee32"}
Jul 16 22:16:43.487355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:43.487252    2555 scope.go:117] "RemoveContainer" containerID="4ead1de9c89ed86402ba1c5d0db50ad04b6d7dba6f461c7c38169a8206b1cdeb"
Jul 16 22:16:43.534909 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:43.534879    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"]
Jul 16 22:16:43.552936 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:43.552910    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xhp9j"]
Jul 16 22:16:44.494196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:16:44.494161    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" path="/var/lib/kubelet/pods/ce78d645-1710-4ba4-a4ce-54dda0e888fa/volumes"
Jul 16 22:17:16.154793 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.154715    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"]
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155032    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155044    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c64eea-b0ea-44de-8b8a-9b133e1ad9d2" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155058    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155064    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155076    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155081    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155123    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155131    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:17:16.155265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.155137    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:17:16.157888 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.157873    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"
Jul 16 22:17:16.160634 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.160616    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:17:16.160733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.160662    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:17:16.165681 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.165650    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"]
Jul 16 22:17:16.293243 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.293189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb87182-3b65-4e7f-aa16-4b0628a0273b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k\" (UID: \"cfb87182-3b65-4e7f-aa16-4b0628a0273b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"
Jul 16 22:17:16.394089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.394051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb87182-3b65-4e7f-aa16-4b0628a0273b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k\" (UID: \"cfb87182-3b65-4e7f-aa16-4b0628a0273b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"
Jul 16 22:17:16.394646 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.394627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb87182-3b65-4e7f-aa16-4b0628a0273b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k\" (UID: \"cfb87182-3b65-4e7f-aa16-4b0628a0273b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"
Jul 16 22:17:16.468891 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.468808    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"
Jul 16 22:17:16.605550 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.605523    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"]
Jul 16 22:17:16.605814 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:17:16.605793    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb87182_3b65_4e7f_aa16_4b0628a0273b.slice/crio-48423f94af3383786149a0d31d3c661b06b38e522c9c301797f6ce0b46367c4b WatchSource:0}: Error finding container 48423f94af3383786149a0d31d3c661b06b38e522c9c301797f6ce0b46367c4b: Status 404 returned error can't find the container with id 48423f94af3383786149a0d31d3c661b06b38e522c9c301797f6ce0b46367c4b
Jul 16 22:17:16.617226 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:16.617203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" event={"ID":"cfb87182-3b65-4e7f-aa16-4b0628a0273b","Type":"ContainerStarted","Data":"48423f94af3383786149a0d31d3c661b06b38e522c9c301797f6ce0b46367c4b"}
Jul 16 22:17:17.622797 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:17.622763    2555 generic.go:356] "Generic (PLEG): container finished" podID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerID="c9f9935edfba5845e86a0a0b66f2cdf196df96dba97ce6d87200fb144aad890c" exitCode=7
Jul 16 22:17:17.623231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:17.622817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" event={"ID":"cfb87182-3b65-4e7f-aa16-4b0628a0273b","Type":"ContainerDied","Data":"c9f9935edfba5845e86a0a0b66f2cdf196df96dba97ce6d87200fb144aad890c"}
Jul 16 22:17:17.623231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:17.623117    2555 scope.go:117] "RemoveContainer" containerID="c9f9935edfba5845e86a0a0b66f2cdf196df96dba97ce6d87200fb144aad890c"
Jul 16 22:17:18.628743 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:18.628712    2555 generic.go:356] "Generic (PLEG): container finished" podID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerID="99bb93a52340f27e9a13a3f4c159687410d64536638c3ba3166e911a62afbf30" exitCode=7
Jul 16 22:17:18.629183 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:18.628804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" event={"ID":"cfb87182-3b65-4e7f-aa16-4b0628a0273b","Type":"ContainerDied","Data":"99bb93a52340f27e9a13a3f4c159687410d64536638c3ba3166e911a62afbf30"}
Jul 16 22:17:18.629183 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:18.628860    2555 scope.go:117] "RemoveContainer" containerID="c9f9935edfba5845e86a0a0b66f2cdf196df96dba97ce6d87200fb144aad890c"
Jul 16 22:17:18.629183 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:18.629119    2555 scope.go:117] "RemoveContainer" containerID="99bb93a52340f27e9a13a3f4c159687410d64536638c3ba3166e911a62afbf30"
Jul 16 22:17:18.629375 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:17:18.629356    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k_odh-ai-gateway-infra(cfb87182-3b65-4e7f-aa16-4b0628a0273b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b"
Jul 16 22:17:19.634496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:19.634465    2555 scope.go:117] "RemoveContainer" containerID="99bb93a52340f27e9a13a3f4c159687410d64536638c3ba3166e911a62afbf30"
Jul 16 22:17:19.634871 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:17:19.634656    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k_odh-ai-gateway-infra(cfb87182-3b65-4e7f-aa16-4b0628a0273b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b"
Jul 16 22:17:33.490148 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:33.490115    2555 scope.go:117] "RemoveContainer" containerID="99bb93a52340f27e9a13a3f4c159687410d64536638c3ba3166e911a62afbf30"
Jul 16 22:17:34.695556 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:34.695523    2555 generic.go:356] "Generic (PLEG): container finished" podID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerID="1d2211a4e3aa197e39dec40057879e80e5ed86e320553be7e25981ba98b0909b" exitCode=7
Jul 16 22:17:34.695949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:34.695579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" event={"ID":"cfb87182-3b65-4e7f-aa16-4b0628a0273b","Type":"ContainerDied","Data":"1d2211a4e3aa197e39dec40057879e80e5ed86e320553be7e25981ba98b0909b"}
Jul 16 22:17:34.695949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:34.695645    2555 scope.go:117] "RemoveContainer" containerID="99bb93a52340f27e9a13a3f4c159687410d64536638c3ba3166e911a62afbf30"
Jul 16 22:17:34.695949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:34.695913    2555 scope.go:117] "RemoveContainer" containerID="1d2211a4e3aa197e39dec40057879e80e5ed86e320553be7e25981ba98b0909b"
Jul 16 22:17:34.696205 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:17:34.696180    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k_odh-ai-gateway-infra(cfb87182-3b65-4e7f-aa16-4b0628a0273b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b"
Jul 16 22:17:35.720419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:35.720383    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"]
Jul 16 22:17:35.848408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:35.848385    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"
Jul 16 22:17:35.963146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:35.963106    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb87182-3b65-4e7f-aa16-4b0628a0273b-maas-api-service-ca\") pod \"cfb87182-3b65-4e7f-aa16-4b0628a0273b\" (UID: \"cfb87182-3b65-4e7f-aa16-4b0628a0273b\") "
Jul 16 22:17:35.963503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:35.963478    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb87182-3b65-4e7f-aa16-4b0628a0273b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfb87182-3b65-4e7f-aa16-4b0628a0273b" (UID: "cfb87182-3b65-4e7f-aa16-4b0628a0273b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:17:36.063757 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:36.063673    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb87182-3b65-4e7f-aa16-4b0628a0273b-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:17:36.707248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:36.707211    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"
Jul 16 22:17:36.707426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:36.707216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k" event={"ID":"cfb87182-3b65-4e7f-aa16-4b0628a0273b","Type":"ContainerDied","Data":"48423f94af3383786149a0d31d3c661b06b38e522c9c301797f6ce0b46367c4b"}
Jul 16 22:17:36.707426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:36.707344    2555 scope.go:117] "RemoveContainer" containerID="1d2211a4e3aa197e39dec40057879e80e5ed86e320553be7e25981ba98b0909b"
Jul 16 22:17:36.723108 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:36.723082    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"]
Jul 16 22:17:36.727077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:36.727054    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-2pt2k"]
Jul 16 22:17:38.494508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:17:38.494478    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" path="/var/lib/kubelet/pods/cfb87182-3b65-4e7f-aa16-4b0628a0273b/volumes"
Jul 16 22:18:04.436213 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436175    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"]
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436466    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436477    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436490    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436496    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce78d645-1710-4ba4-a4ce-54dda0e888fa" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436508    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436514    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436576    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436584    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:04.436723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.436590    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:04.439531 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.439509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"
Jul 16 22:18:04.442255 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.442230    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:18:04.442255 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.442243    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:18:04.450490 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.450461    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"]
Jul 16 22:18:04.615391 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.615360    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc5753-da50-4f66-8e7f-623613bd040a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc\" (UID: \"04cc5753-da50-4f66-8e7f-623613bd040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"
Jul 16 22:18:04.716805 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.716712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc5753-da50-4f66-8e7f-623613bd040a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc\" (UID: \"04cc5753-da50-4f66-8e7f-623613bd040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"
Jul 16 22:18:04.717353 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.717335    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc5753-da50-4f66-8e7f-623613bd040a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc\" (UID: \"04cc5753-da50-4f66-8e7f-623613bd040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"
Jul 16 22:18:04.751411 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.751377    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"
Jul 16 22:18:04.880886 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:04.880863    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"]
Jul 16 22:18:04.883510 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:18:04.883487    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cc5753_da50_4f66_8e7f_623613bd040a.slice/crio-047c1272064a26df401498b1b19ab43ea041663ace6a559623c54a2bb4ff0541 WatchSource:0}: Error finding container 047c1272064a26df401498b1b19ab43ea041663ace6a559623c54a2bb4ff0541: Status 404 returned error can't find the container with id 047c1272064a26df401498b1b19ab43ea041663ace6a559623c54a2bb4ff0541
Jul 16 22:18:05.835325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:05.835293    2555 generic.go:356] "Generic (PLEG): container finished" podID="04cc5753-da50-4f66-8e7f-623613bd040a" containerID="e23552ea8ecbc15eea39f91839f2d6e9d4106b29a23453b2ee4f2ab87d9ef5a4" exitCode=7
Jul 16 22:18:05.835800 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:05.835380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" event={"ID":"04cc5753-da50-4f66-8e7f-623613bd040a","Type":"ContainerDied","Data":"e23552ea8ecbc15eea39f91839f2d6e9d4106b29a23453b2ee4f2ab87d9ef5a4"}
Jul 16 22:18:05.835800 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:05.835463    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" event={"ID":"04cc5753-da50-4f66-8e7f-623613bd040a","Type":"ContainerStarted","Data":"047c1272064a26df401498b1b19ab43ea041663ace6a559623c54a2bb4ff0541"}
Jul 16 22:18:05.835800 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:05.835585    2555 scope.go:117] "RemoveContainer" containerID="e23552ea8ecbc15eea39f91839f2d6e9d4106b29a23453b2ee4f2ab87d9ef5a4"
Jul 16 22:18:06.841023 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:06.840979    2555 generic.go:356] "Generic (PLEG): container finished" podID="04cc5753-da50-4f66-8e7f-623613bd040a" containerID="cef422e7c782e221eb638fc20dcf8cd253c3688a60c4127b5f3215e17ea9e028" exitCode=7
Jul 16 22:18:06.841441 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:06.841038    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" event={"ID":"04cc5753-da50-4f66-8e7f-623613bd040a","Type":"ContainerDied","Data":"cef422e7c782e221eb638fc20dcf8cd253c3688a60c4127b5f3215e17ea9e028"}
Jul 16 22:18:06.841441 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:06.841094    2555 scope.go:117] "RemoveContainer" containerID="e23552ea8ecbc15eea39f91839f2d6e9d4106b29a23453b2ee4f2ab87d9ef5a4"
Jul 16 22:18:06.841441 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:06.841282    2555 scope.go:117] "RemoveContainer" containerID="cef422e7c782e221eb638fc20dcf8cd253c3688a60c4127b5f3215e17ea9e028"
Jul 16 22:18:06.841561 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:06.841476    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc_odh-ai-gateway-infra(04cc5753-da50-4f66-8e7f-623613bd040a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" podUID="04cc5753-da50-4f66-8e7f-623613bd040a"
Jul 16 22:18:07.846801 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:07.846771    2555 scope.go:117] "RemoveContainer" containerID="cef422e7c782e221eb638fc20dcf8cd253c3688a60c4127b5f3215e17ea9e028"
Jul 16 22:18:07.847182 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:07.847005    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc_odh-ai-gateway-infra(04cc5753-da50-4f66-8e7f-623613bd040a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" podUID="04cc5753-da50-4f66-8e7f-623613bd040a"
Jul 16 22:18:16.201757 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.201717    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"]
Jul 16 22:18:16.202313 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.202188    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:16.202313 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.202207    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb87182-3b65-4e7f-aa16-4b0628a0273b" containerName="revoke-keys"
Jul 16 22:18:16.206418 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.206394    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"
Jul 16 22:18:16.214875 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.214853    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"]
Jul 16 22:18:16.314285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.314251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94120e1b-bf18-44fc-bb9a-673a21a68379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh\" (UID: \"94120e1b-bf18-44fc-bb9a-673a21a68379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"
Jul 16 22:18:16.415281 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.415232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94120e1b-bf18-44fc-bb9a-673a21a68379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh\" (UID: \"94120e1b-bf18-44fc-bb9a-673a21a68379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"
Jul 16 22:18:16.415837 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.415816    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94120e1b-bf18-44fc-bb9a-673a21a68379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh\" (UID: \"94120e1b-bf18-44fc-bb9a-673a21a68379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"
Jul 16 22:18:16.517861 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.517781    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"
Jul 16 22:18:16.665071 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.665037    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"]
Jul 16 22:18:16.668813 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:18:16.668786    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94120e1b_bf18_44fc_bb9a_673a21a68379.slice/crio-e41e9d210292b5c623861046502998f6738fa8499e22d6fc69e6227d50a72d37 WatchSource:0}: Error finding container e41e9d210292b5c623861046502998f6738fa8499e22d6fc69e6227d50a72d37: Status 404 returned error can't find the container with id e41e9d210292b5c623861046502998f6738fa8499e22d6fc69e6227d50a72d37
Jul 16 22:18:16.890444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:16.890411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" event={"ID":"94120e1b-bf18-44fc-bb9a-673a21a68379","Type":"ContainerStarted","Data":"e41e9d210292b5c623861046502998f6738fa8499e22d6fc69e6227d50a72d37"}
Jul 16 22:18:17.896403 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:17.896369    2555 generic.go:356] "Generic (PLEG): container finished" podID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerID="9f9846f0b2ba61c06af428f16d48edbc9b68384f36bf82a2b0ececf38c29668e" exitCode=6
Jul 16 22:18:17.896791 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:17.896410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" event={"ID":"94120e1b-bf18-44fc-bb9a-673a21a68379","Type":"ContainerDied","Data":"9f9846f0b2ba61c06af428f16d48edbc9b68384f36bf82a2b0ececf38c29668e"}
Jul 16 22:18:17.896791 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:17.896634    2555 scope.go:117] "RemoveContainer" containerID="9f9846f0b2ba61c06af428f16d48edbc9b68384f36bf82a2b0ececf38c29668e"
Jul 16 22:18:18.902089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:18.902056    2555 generic.go:356] "Generic (PLEG): container finished" podID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerID="4c099996812fc2a7ec5cdbbcfb7a3aa42dcae116c689a2b9b24be70aeb1a2e72" exitCode=6
Jul 16 22:18:18.902497 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:18.902139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" event={"ID":"94120e1b-bf18-44fc-bb9a-673a21a68379","Type":"ContainerDied","Data":"4c099996812fc2a7ec5cdbbcfb7a3aa42dcae116c689a2b9b24be70aeb1a2e72"}
Jul 16 22:18:18.902497 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:18.902202    2555 scope.go:117] "RemoveContainer" containerID="9f9846f0b2ba61c06af428f16d48edbc9b68384f36bf82a2b0ececf38c29668e"
Jul 16 22:18:18.902497 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:18.902314    2555 scope.go:117] "RemoveContainer" containerID="4c099996812fc2a7ec5cdbbcfb7a3aa42dcae116c689a2b9b24be70aeb1a2e72"
Jul 16 22:18:18.902613 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:18.902530    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh_odh-ai-gateway-infra(94120e1b-bf18-44fc-bb9a-673a21a68379)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379"
Jul 16 22:18:19.908104 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:19.908080    2555 scope.go:117] "RemoveContainer" containerID="4c099996812fc2a7ec5cdbbcfb7a3aa42dcae116c689a2b9b24be70aeb1a2e72"
Jul 16 22:18:19.908506 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:19.908273    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh_odh-ai-gateway-infra(94120e1b-bf18-44fc-bb9a-673a21a68379)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379"
Jul 16 22:18:20.489858 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:20.489826    2555 scope.go:117] "RemoveContainer" containerID="cef422e7c782e221eb638fc20dcf8cd253c3688a60c4127b5f3215e17ea9e028"
Jul 16 22:18:20.914329 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:20.914294    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" event={"ID":"04cc5753-da50-4f66-8e7f-623613bd040a","Type":"ContainerStarted","Data":"60e327c94225a60031806608a74be9dbc95c6d8caa63122767f8a72aabb0dcc3"}
Jul 16 22:18:20.932175 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:20.932130    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" podStartSLOduration=16.932114616 podStartE2EDuration="16.932114616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:20.930744926 +0000 UTC m=+1856.923909078" watchObservedRunningTime="2026-07-16 22:18:20.932114616 +0000 UTC m=+1856.925278781"
Jul 16 22:18:21.919911 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:21.919879    2555 generic.go:356] "Generic (PLEG): container finished" podID="04cc5753-da50-4f66-8e7f-623613bd040a" containerID="60e327c94225a60031806608a74be9dbc95c6d8caa63122767f8a72aabb0dcc3" exitCode=7
Jul 16 22:18:21.920456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:21.919974    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" event={"ID":"04cc5753-da50-4f66-8e7f-623613bd040a","Type":"ContainerDied","Data":"60e327c94225a60031806608a74be9dbc95c6d8caa63122767f8a72aabb0dcc3"}
Jul 16 22:18:21.920456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:21.920033    2555 scope.go:117] "RemoveContainer" containerID="cef422e7c782e221eb638fc20dcf8cd253c3688a60c4127b5f3215e17ea9e028"
Jul 16 22:18:21.920456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:21.920352    2555 scope.go:117] "RemoveContainer" containerID="60e327c94225a60031806608a74be9dbc95c6d8caa63122767f8a72aabb0dcc3"
Jul 16 22:18:21.920620 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:21.920584    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc_odh-ai-gateway-infra(04cc5753-da50-4f66-8e7f-623613bd040a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" podUID="04cc5753-da50-4f66-8e7f-623613bd040a"
Jul 16 22:18:21.944972 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:21.944922    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"]
Jul 16 22:18:23.062405 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.062377    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"
Jul 16 22:18:23.178339 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.178259    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc5753-da50-4f66-8e7f-623613bd040a-maas-api-service-ca\") pod \"04cc5753-da50-4f66-8e7f-623613bd040a\" (UID: \"04cc5753-da50-4f66-8e7f-623613bd040a\") "
Jul 16 22:18:23.178612 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.178589    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cc5753-da50-4f66-8e7f-623613bd040a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04cc5753-da50-4f66-8e7f-623613bd040a" (UID: "04cc5753-da50-4f66-8e7f-623613bd040a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:18:23.279412 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.279374    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cc5753-da50-4f66-8e7f-623613bd040a-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:18:23.932101 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.932066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc" event={"ID":"04cc5753-da50-4f66-8e7f-623613bd040a","Type":"ContainerDied","Data":"047c1272064a26df401498b1b19ab43ea041663ace6a559623c54a2bb4ff0541"}
Jul 16 22:18:23.932101 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.932088    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"
Jul 16 22:18:23.932327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.932129    2555 scope.go:117] "RemoveContainer" containerID="60e327c94225a60031806608a74be9dbc95c6d8caa63122767f8a72aabb0dcc3"
Jul 16 22:18:23.955686 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.955656    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"]
Jul 16 22:18:23.959799 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:23.959776    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-wnzhc"]
Jul 16 22:18:24.494929 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:24.494892    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" path="/var/lib/kubelet/pods/04cc5753-da50-4f66-8e7f-623613bd040a/volumes"
Jul 16 22:18:33.490163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:33.490134    2555 scope.go:117] "RemoveContainer" containerID="4c099996812fc2a7ec5cdbbcfb7a3aa42dcae116c689a2b9b24be70aeb1a2e72"
Jul 16 22:18:33.982072 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:33.982040    2555 generic.go:356] "Generic (PLEG): container finished" podID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerID="a3b5ab807bf42a75f252fbcf80d18ecee642cd18d6e47e87e5f1a2187601654d" exitCode=6
Jul 16 22:18:33.982253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:33.982117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" event={"ID":"94120e1b-bf18-44fc-bb9a-673a21a68379","Type":"ContainerDied","Data":"a3b5ab807bf42a75f252fbcf80d18ecee642cd18d6e47e87e5f1a2187601654d"}
Jul 16 22:18:33.982253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:33.982177    2555 scope.go:117] "RemoveContainer" containerID="4c099996812fc2a7ec5cdbbcfb7a3aa42dcae116c689a2b9b24be70aeb1a2e72"
Jul 16 22:18:33.982487 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:33.982472    2555 scope.go:117] "RemoveContainer" containerID="a3b5ab807bf42a75f252fbcf80d18ecee642cd18d6e47e87e5f1a2187601654d"
Jul 16 22:18:33.982710 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:33.982689    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh_odh-ai-gateway-infra(94120e1b-bf18-44fc-bb9a-673a21a68379)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379"
Jul 16 22:18:35.007775 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.007734    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"]
Jul 16 22:18:35.136785 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.136761    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"
Jul 16 22:18:35.277456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.277366    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94120e1b-bf18-44fc-bb9a-673a21a68379-maas-api-service-ca\") pod \"94120e1b-bf18-44fc-bb9a-673a21a68379\" (UID: \"94120e1b-bf18-44fc-bb9a-673a21a68379\") "
Jul 16 22:18:35.277796 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.277768    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94120e1b-bf18-44fc-bb9a-673a21a68379-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94120e1b-bf18-44fc-bb9a-673a21a68379" (UID: "94120e1b-bf18-44fc-bb9a-673a21a68379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:18:35.378413 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.378369    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94120e1b-bf18-44fc-bb9a-673a21a68379-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:18:35.997804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.997774    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"
Jul 16 22:18:35.997804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.997783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh" event={"ID":"94120e1b-bf18-44fc-bb9a-673a21a68379","Type":"ContainerDied","Data":"e41e9d210292b5c623861046502998f6738fa8499e22d6fc69e6227d50a72d37"}
Jul 16 22:18:35.998074 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:35.997839    2555 scope.go:117] "RemoveContainer" containerID="a3b5ab807bf42a75f252fbcf80d18ecee642cd18d6e47e87e5f1a2187601654d"
Jul 16 22:18:36.020426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:36.020396    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"]
Jul 16 22:18:36.028178 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:36.028154    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-g6rvh"]
Jul 16 22:18:36.494067 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:36.494038    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" path="/var/lib/kubelet/pods/94120e1b-bf18-44fc-bb9a-673a21a68379/volumes"
Jul 16 22:18:54.485258 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485201    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"]
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485525    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485538    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485554    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485559    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485566    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485572    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485584    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485590    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485650    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485658    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485666    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485671    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485677    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:18:54.485739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.485693    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:18:54.491447 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.491421    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"
Jul 16 22:18:54.494678 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.494649    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:18:54.494678 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.494649    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:18:54.498652 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.498626    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"]
Jul 16 22:18:54.638912 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.638875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75878d7-f05e-418d-bf94-03d4ed858969-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-74bdm\" (UID: \"c75878d7-f05e-418d-bf94-03d4ed858969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"
Jul 16 22:18:54.739774 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.739698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75878d7-f05e-418d-bf94-03d4ed858969-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-74bdm\" (UID: \"c75878d7-f05e-418d-bf94-03d4ed858969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"
Jul 16 22:18:54.740327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.740307    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75878d7-f05e-418d-bf94-03d4ed858969-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-74bdm\" (UID: \"c75878d7-f05e-418d-bf94-03d4ed858969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"
Jul 16 22:18:54.805550 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.805522    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"
Jul 16 22:18:54.954006 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:54.953978    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"]
Jul 16 22:18:54.956649 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:18:54.956621    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75878d7_f05e_418d_bf94_03d4ed858969.slice/crio-a8946eb68cadf29deed753e0da53bdb534476c3dcfb19c067a5a0e0dcfd2f7a7 WatchSource:0}: Error finding container a8946eb68cadf29deed753e0da53bdb534476c3dcfb19c067a5a0e0dcfd2f7a7: Status 404 returned error can't find the container with id a8946eb68cadf29deed753e0da53bdb534476c3dcfb19c067a5a0e0dcfd2f7a7
Jul 16 22:18:55.080463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:55.080429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" event={"ID":"c75878d7-f05e-418d-bf94-03d4ed858969","Type":"ContainerStarted","Data":"a8946eb68cadf29deed753e0da53bdb534476c3dcfb19c067a5a0e0dcfd2f7a7"}
Jul 16 22:18:56.086131 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:56.086096    2555 generic.go:356] "Generic (PLEG): container finished" podID="c75878d7-f05e-418d-bf94-03d4ed858969" containerID="c2ebcc91bb1f7d85693807102452e9874290e0d31e0a32ae65e17200d371eefe" exitCode=7
Jul 16 22:18:56.086511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:56.086138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" event={"ID":"c75878d7-f05e-418d-bf94-03d4ed858969","Type":"ContainerDied","Data":"c2ebcc91bb1f7d85693807102452e9874290e0d31e0a32ae65e17200d371eefe"}
Jul 16 22:18:56.086511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:56.086376    2555 scope.go:117] "RemoveContainer" containerID="c2ebcc91bb1f7d85693807102452e9874290e0d31e0a32ae65e17200d371eefe"
Jul 16 22:18:57.091422 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:57.091388    2555 generic.go:356] "Generic (PLEG): container finished" podID="c75878d7-f05e-418d-bf94-03d4ed858969" containerID="5160f94981d5774e5ca4aefe25927a0d67caca405fd969d1773a0131cd438302" exitCode=7
Jul 16 22:18:57.091816 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:57.091465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" event={"ID":"c75878d7-f05e-418d-bf94-03d4ed858969","Type":"ContainerDied","Data":"5160f94981d5774e5ca4aefe25927a0d67caca405fd969d1773a0131cd438302"}
Jul 16 22:18:57.091816 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:57.091518    2555 scope.go:117] "RemoveContainer" containerID="c2ebcc91bb1f7d85693807102452e9874290e0d31e0a32ae65e17200d371eefe"
Jul 16 22:18:57.091816 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:57.091714    2555 scope.go:117] "RemoveContainer" containerID="5160f94981d5774e5ca4aefe25927a0d67caca405fd969d1773a0131cd438302"
Jul 16 22:18:57.091984 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:57.091949    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-74bdm_odh-ai-gateway-infra(c75878d7-f05e-418d-bf94-03d4ed858969)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" podUID="c75878d7-f05e-418d-bf94-03d4ed858969"
Jul 16 22:18:58.097294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:18:58.097265    2555 scope.go:117] "RemoveContainer" containerID="5160f94981d5774e5ca4aefe25927a0d67caca405fd969d1773a0131cd438302"
Jul 16 22:18:58.097691 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:18:58.097436    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-74bdm_odh-ai-gateway-infra(c75878d7-f05e-418d-bf94-03d4ed858969)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" podUID="c75878d7-f05e-418d-bf94-03d4ed858969"
Jul 16 22:19:06.265066 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.265030    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"]
Jul 16 22:19:06.265453 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.265321    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:19:06.265453 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.265332    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94120e1b-bf18-44fc-bb9a-673a21a68379" containerName="revoke-keys"
Jul 16 22:19:06.265453 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.265351    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:19:06.265453 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.265357    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cc5753-da50-4f66-8e7f-623613bd040a" containerName="revoke-keys"
Jul 16 22:19:06.269528 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.269511    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"
Jul 16 22:19:06.287916 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.287885    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"]
Jul 16 22:19:06.335244 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.335204    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8289a920-84f8-411b-b65b-677da680ac13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh\" (UID: \"8289a920-84f8-411b-b65b-677da680ac13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"
Jul 16 22:19:06.436578 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.436527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8289a920-84f8-411b-b65b-677da680ac13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh\" (UID: \"8289a920-84f8-411b-b65b-677da680ac13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"
Jul 16 22:19:06.437190 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.437172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8289a920-84f8-411b-b65b-677da680ac13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh\" (UID: \"8289a920-84f8-411b-b65b-677da680ac13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"
Jul 16 22:19:06.578850 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.578813    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"
Jul 16 22:19:06.728948 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:06.728921    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"]
Jul 16 22:19:06.730470 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:19:06.730446    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8289a920_84f8_411b_b65b_677da680ac13.slice/crio-f57fa1be1c055c4ea6ff0e32d7e9ac43e545315237d6b46727e86f1ca796b872 WatchSource:0}: Error finding container f57fa1be1c055c4ea6ff0e32d7e9ac43e545315237d6b46727e86f1ca796b872: Status 404 returned error can't find the container with id f57fa1be1c055c4ea6ff0e32d7e9ac43e545315237d6b46727e86f1ca796b872
Jul 16 22:19:07.136312 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:07.136274    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" event={"ID":"8289a920-84f8-411b-b65b-677da680ac13","Type":"ContainerStarted","Data":"f9e84d3439708d83829632765007cd5231ab66f8fe7a3f403dad64ed521a674e"}
Jul 16 22:19:07.136489 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:07.136339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" event={"ID":"8289a920-84f8-411b-b65b-677da680ac13","Type":"ContainerStarted","Data":"f57fa1be1c055c4ea6ff0e32d7e9ac43e545315237d6b46727e86f1ca796b872"}
Jul 16 22:19:07.161882 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:07.161827    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" podStartSLOduration=1.161808304 podStartE2EDuration="1.161808304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:07.160723683 +0000 UTC m=+1903.153887834" watchObservedRunningTime="2026-07-16 22:19:07.161808304 +0000 UTC m=+1903.154972466"
Jul 16 22:19:08.142666 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:08.142633    2555 generic.go:356] "Generic (PLEG): container finished" podID="8289a920-84f8-411b-b65b-677da680ac13" containerID="f9e84d3439708d83829632765007cd5231ab66f8fe7a3f403dad64ed521a674e" exitCode=6
Jul 16 22:19:08.143179 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:08.142732    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" event={"ID":"8289a920-84f8-411b-b65b-677da680ac13","Type":"ContainerDied","Data":"f9e84d3439708d83829632765007cd5231ab66f8fe7a3f403dad64ed521a674e"}
Jul 16 22:19:08.143179 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:08.142944    2555 scope.go:117] "RemoveContainer" containerID="f9e84d3439708d83829632765007cd5231ab66f8fe7a3f403dad64ed521a674e"
Jul 16 22:19:09.148305 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:09.148267    2555 generic.go:356] "Generic (PLEG): container finished" podID="8289a920-84f8-411b-b65b-677da680ac13" containerID="ffbe0d6d23f876e6f0a24351ef3c59671888a8ee974fbd9d2e5a26281319624a" exitCode=6
Jul 16 22:19:09.148777 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:09.148362    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" event={"ID":"8289a920-84f8-411b-b65b-677da680ac13","Type":"ContainerDied","Data":"ffbe0d6d23f876e6f0a24351ef3c59671888a8ee974fbd9d2e5a26281319624a"}
Jul 16 22:19:09.148777 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:09.148486    2555 scope.go:117] "RemoveContainer" containerID="f9e84d3439708d83829632765007cd5231ab66f8fe7a3f403dad64ed521a674e"
Jul 16 22:19:09.148777 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:09.148712    2555 scope.go:117] "RemoveContainer" containerID="ffbe0d6d23f876e6f0a24351ef3c59671888a8ee974fbd9d2e5a26281319624a"
Jul 16 22:19:09.148914 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:09.148897    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh_odh-ai-gateway-infra(8289a920-84f8-411b-b65b-677da680ac13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" podUID="8289a920-84f8-411b-b65b-677da680ac13"
Jul 16 22:19:10.155435 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:10.155407    2555 scope.go:117] "RemoveContainer" containerID="ffbe0d6d23f876e6f0a24351ef3c59671888a8ee974fbd9d2e5a26281319624a"
Jul 16 22:19:10.155836 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:10.155596    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh_odh-ai-gateway-infra(8289a920-84f8-411b-b65b-677da680ac13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" podUID="8289a920-84f8-411b-b65b-677da680ac13"
Jul 16 22:19:13.490305 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:13.490276    2555 scope.go:117] "RemoveContainer" containerID="5160f94981d5774e5ca4aefe25927a0d67caca405fd969d1773a0131cd438302"
Jul 16 22:19:14.175554 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:14.175519    2555 generic.go:356] "Generic (PLEG): container finished" podID="c75878d7-f05e-418d-bf94-03d4ed858969" containerID="50a7be125309240567c5c7b771e62616a49e96b48783bdb739ade6df8e6d1000" exitCode=6
Jul 16 22:19:14.175688 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:14.175592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" event={"ID":"c75878d7-f05e-418d-bf94-03d4ed858969","Type":"ContainerDied","Data":"50a7be125309240567c5c7b771e62616a49e96b48783bdb739ade6df8e6d1000"}
Jul 16 22:19:14.175688 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:14.175651    2555 scope.go:117] "RemoveContainer" containerID="5160f94981d5774e5ca4aefe25927a0d67caca405fd969d1773a0131cd438302"
Jul 16 22:19:14.175952 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:14.175926    2555 scope.go:117] "RemoveContainer" containerID="50a7be125309240567c5c7b771e62616a49e96b48783bdb739ade6df8e6d1000"
Jul 16 22:19:14.176191 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:14.176166    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-74bdm_odh-ai-gateway-infra(c75878d7-f05e-418d-bf94-03d4ed858969)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" podUID="c75878d7-f05e-418d-bf94-03d4ed858969"
Jul 16 22:19:15.233577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:15.233542    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"]
Jul 16 22:19:15.366702 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:15.366676    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"
Jul 16 22:19:15.413236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:15.413203    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75878d7-f05e-418d-bf94-03d4ed858969-maas-api-service-ca\") pod \"c75878d7-f05e-418d-bf94-03d4ed858969\" (UID: \"c75878d7-f05e-418d-bf94-03d4ed858969\") "
Jul 16 22:19:15.413568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:15.413545    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75878d7-f05e-418d-bf94-03d4ed858969-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c75878d7-f05e-418d-bf94-03d4ed858969" (UID: "c75878d7-f05e-418d-bf94-03d4ed858969"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:15.514497 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:15.514408    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75878d7-f05e-418d-bf94-03d4ed858969-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:19:16.186597 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:16.186568    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"
Jul 16 22:19:16.186597 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:16.186576    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm" event={"ID":"c75878d7-f05e-418d-bf94-03d4ed858969","Type":"ContainerDied","Data":"a8946eb68cadf29deed753e0da53bdb534476c3dcfb19c067a5a0e0dcfd2f7a7"}
Jul 16 22:19:16.186837 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:16.186631    2555 scope.go:117] "RemoveContainer" containerID="50a7be125309240567c5c7b771e62616a49e96b48783bdb739ade6df8e6d1000"
Jul 16 22:19:16.218805 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:16.218776    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"]
Jul 16 22:19:16.223389 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:16.223364    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-74bdm"]
Jul 16 22:19:16.494288 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:16.494216    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" path="/var/lib/kubelet/pods/c75878d7-f05e-418d-bf94-03d4ed858969/volumes"
Jul 16 22:19:21.489631 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:21.489605    2555 scope.go:117] "RemoveContainer" containerID="ffbe0d6d23f876e6f0a24351ef3c59671888a8ee974fbd9d2e5a26281319624a"
Jul 16 22:19:22.217914 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:22.217884    2555 generic.go:356] "Generic (PLEG): container finished" podID="8289a920-84f8-411b-b65b-677da680ac13" containerID="6a3e10408a66c70c1d4e5d829deffe7bf24bc0ee710d349bb131b7dbd7479bfe" exitCode=6
Jul 16 22:19:22.218058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:22.217933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" event={"ID":"8289a920-84f8-411b-b65b-677da680ac13","Type":"ContainerDied","Data":"6a3e10408a66c70c1d4e5d829deffe7bf24bc0ee710d349bb131b7dbd7479bfe"}
Jul 16 22:19:22.218058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:22.217999    2555 scope.go:117] "RemoveContainer" containerID="ffbe0d6d23f876e6f0a24351ef3c59671888a8ee974fbd9d2e5a26281319624a"
Jul 16 22:19:22.218277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:22.218254    2555 scope.go:117] "RemoveContainer" containerID="6a3e10408a66c70c1d4e5d829deffe7bf24bc0ee710d349bb131b7dbd7479bfe"
Jul 16 22:19:22.218515 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:22.218495    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh_odh-ai-gateway-infra(8289a920-84f8-411b-b65b-677da680ac13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" podUID="8289a920-84f8-411b-b65b-677da680ac13"
Jul 16 22:19:23.292571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:23.292529    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"]
Jul 16 22:19:23.422567 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:23.422544    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"
Jul 16 22:19:23.489047 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:23.489016    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8289a920-84f8-411b-b65b-677da680ac13-maas-api-service-ca\") pod \"8289a920-84f8-411b-b65b-677da680ac13\" (UID: \"8289a920-84f8-411b-b65b-677da680ac13\") "
Jul 16 22:19:23.489409 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:23.489379    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8289a920-84f8-411b-b65b-677da680ac13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8289a920-84f8-411b-b65b-677da680ac13" (UID: "8289a920-84f8-411b-b65b-677da680ac13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:23.590196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:23.590163    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8289a920-84f8-411b-b65b-677da680ac13-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:19:24.228410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.228382    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"
Jul 16 22:19:24.228580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.228369    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh" event={"ID":"8289a920-84f8-411b-b65b-677da680ac13","Type":"ContainerDied","Data":"f57fa1be1c055c4ea6ff0e32d7e9ac43e545315237d6b46727e86f1ca796b872"}
Jul 16 22:19:24.228580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.228525    2555 scope.go:117] "RemoveContainer" containerID="6a3e10408a66c70c1d4e5d829deffe7bf24bc0ee710d349bb131b7dbd7479bfe"
Jul 16 22:19:24.286353 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.286321    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"]
Jul 16 22:19:24.290310 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.290267    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7wzgh"]
Jul 16 22:19:24.495805 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.495730    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8289a920-84f8-411b-b65b-677da680ac13" path="/var/lib/kubelet/pods/8289a920-84f8-411b-b65b-677da680ac13/volumes"
Jul 16 22:19:24.889372 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889339    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"]
Jul 16 22:19:24.889706 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889682    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:24.889803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889709    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:24.889803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889723    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:24.889803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889731    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:24.889803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889759    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:24.889803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889767    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:24.889803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889786    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:24.889803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889795    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:24.890140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889886    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:24.890140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889899    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:24.890140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889911    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:24.890140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889922    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:24.890140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.889931    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:24.894289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.894269    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"
Jul 16 22:19:24.898670 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.898650    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:19:24.898767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.898654    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:19:24.921230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:24.921201    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"]
Jul 16 22:19:25.002673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:25.002641    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e138b-cf54-4ecb-a819-4e44305c2af3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644\" (UID: \"536e138b-cf54-4ecb-a819-4e44305c2af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"
Jul 16 22:19:25.103545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:25.103507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e138b-cf54-4ecb-a819-4e44305c2af3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644\" (UID: \"536e138b-cf54-4ecb-a819-4e44305c2af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"
Jul 16 22:19:25.104140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:25.104122    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e138b-cf54-4ecb-a819-4e44305c2af3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644\" (UID: \"536e138b-cf54-4ecb-a819-4e44305c2af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"
Jul 16 22:19:25.204121 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:25.204056    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"
Jul 16 22:19:25.353511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:25.353486    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"]
Jul 16 22:19:25.355530 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:19:25.355504    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536e138b_cf54_4ecb_a819_4e44305c2af3.slice/crio-e6ee69df6a6191577b1f18c7ad58a50df69de66d2d6ac1858dd65cf048893c16 WatchSource:0}: Error finding container e6ee69df6a6191577b1f18c7ad58a50df69de66d2d6ac1858dd65cf048893c16: Status 404 returned error can't find the container with id e6ee69df6a6191577b1f18c7ad58a50df69de66d2d6ac1858dd65cf048893c16
Jul 16 22:19:26.238518 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:26.238483    2555 generic.go:356] "Generic (PLEG): container finished" podID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerID="5bf279d8d842caa9c39d5ae31bf5f11b966b9240217a2ad80b94d983257fa2b4" exitCode=7
Jul 16 22:19:26.239002 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:26.238570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" event={"ID":"536e138b-cf54-4ecb-a819-4e44305c2af3","Type":"ContainerDied","Data":"5bf279d8d842caa9c39d5ae31bf5f11b966b9240217a2ad80b94d983257fa2b4"}
Jul 16 22:19:26.239002 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:26.238634    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" event={"ID":"536e138b-cf54-4ecb-a819-4e44305c2af3","Type":"ContainerStarted","Data":"e6ee69df6a6191577b1f18c7ad58a50df69de66d2d6ac1858dd65cf048893c16"}
Jul 16 22:19:26.239002 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:26.238778    2555 scope.go:117] "RemoveContainer" containerID="5bf279d8d842caa9c39d5ae31bf5f11b966b9240217a2ad80b94d983257fa2b4"
Jul 16 22:19:27.244785 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:27.244698    2555 generic.go:356] "Generic (PLEG): container finished" podID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerID="2bd968450febdc53240bb0cff1476e8762f3ebe48ab6ebdaf2570be23b9a5264" exitCode=7
Jul 16 22:19:27.245231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:27.244798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" event={"ID":"536e138b-cf54-4ecb-a819-4e44305c2af3","Type":"ContainerDied","Data":"2bd968450febdc53240bb0cff1476e8762f3ebe48ab6ebdaf2570be23b9a5264"}
Jul 16 22:19:27.245231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:27.244871    2555 scope.go:117] "RemoveContainer" containerID="5bf279d8d842caa9c39d5ae31bf5f11b966b9240217a2ad80b94d983257fa2b4"
Jul 16 22:19:27.245231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:27.244979    2555 scope.go:117] "RemoveContainer" containerID="2bd968450febdc53240bb0cff1476e8762f3ebe48ab6ebdaf2570be23b9a5264"
Jul 16 22:19:27.245231 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:27.245197    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644_odh-ai-gateway-infra(536e138b-cf54-4ecb-a819-4e44305c2af3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3"
Jul 16 22:19:28.251998 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:28.251942    2555 scope.go:117] "RemoveContainer" containerID="2bd968450febdc53240bb0cff1476e8762f3ebe48ab6ebdaf2570be23b9a5264"
Jul 16 22:19:28.252389 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:28.252175    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644_odh-ai-gateway-infra(536e138b-cf54-4ecb-a819-4e44305c2af3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3"
Jul 16 22:19:40.490068 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:40.490030    2555 scope.go:117] "RemoveContainer" containerID="2bd968450febdc53240bb0cff1476e8762f3ebe48ab6ebdaf2570be23b9a5264"
Jul 16 22:19:41.307227 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:41.307145    2555 generic.go:356] "Generic (PLEG): container finished" podID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerID="c2d26a327a73350f0736e8b27342c64d8509fdc0c5ffb0b60ace01383a42264a" exitCode=7
Jul 16 22:19:41.307227 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:41.307221    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" event={"ID":"536e138b-cf54-4ecb-a819-4e44305c2af3","Type":"ContainerDied","Data":"c2d26a327a73350f0736e8b27342c64d8509fdc0c5ffb0b60ace01383a42264a"}
Jul 16 22:19:41.307414 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:41.307284    2555 scope.go:117] "RemoveContainer" containerID="2bd968450febdc53240bb0cff1476e8762f3ebe48ab6ebdaf2570be23b9a5264"
Jul 16 22:19:41.307568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:41.307554    2555 scope.go:117] "RemoveContainer" containerID="c2d26a327a73350f0736e8b27342c64d8509fdc0c5ffb0b60ace01383a42264a"
Jul 16 22:19:41.307791 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:41.307771    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644_odh-ai-gateway-infra(536e138b-cf54-4ecb-a819-4e44305c2af3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3"
Jul 16 22:19:42.334449 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:42.334415    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"]
Jul 16 22:19:42.461723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:42.461694    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"
Jul 16 22:19:42.562947 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:42.562911    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e138b-cf54-4ecb-a819-4e44305c2af3-maas-api-service-ca\") pod \"536e138b-cf54-4ecb-a819-4e44305c2af3\" (UID: \"536e138b-cf54-4ecb-a819-4e44305c2af3\") "
Jul 16 22:19:42.563263 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:42.563241    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536e138b-cf54-4ecb-a819-4e44305c2af3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "536e138b-cf54-4ecb-a819-4e44305c2af3" (UID: "536e138b-cf54-4ecb-a819-4e44305c2af3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:42.664337 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:42.664254    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e138b-cf54-4ecb-a819-4e44305c2af3-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:19:43.317322 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:43.317289    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"
Jul 16 22:19:43.317516 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:43.317320    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644" event={"ID":"536e138b-cf54-4ecb-a819-4e44305c2af3","Type":"ContainerDied","Data":"e6ee69df6a6191577b1f18c7ad58a50df69de66d2d6ac1858dd65cf048893c16"}
Jul 16 22:19:43.317516 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:43.317385    2555 scope.go:117] "RemoveContainer" containerID="c2d26a327a73350f0736e8b27342c64d8509fdc0c5ffb0b60ace01383a42264a"
Jul 16 22:19:43.357994 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:43.357939    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"]
Jul 16 22:19:43.371387 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:43.371359    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-z6644"]
Jul 16 22:19:44.496061 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:44.496026    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" path="/var/lib/kubelet/pods/536e138b-cf54-4ecb-a819-4e44305c2af3/volumes"
Jul 16 22:19:54.546067 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546033    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"]
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546340    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546351    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546362    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546370    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75878d7-f05e-418d-bf94-03d4ed858969" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546381    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546387    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546393    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546399    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546404    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546409    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546463    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546470    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8289a920-84f8-411b-b65b-677da680ac13" containerName="revoke-keys"
Jul 16 22:19:54.546472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546478    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.546846 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.546483    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="536e138b-cf54-4ecb-a819-4e44305c2af3" containerName="revoke-keys"
Jul 16 22:19:54.550700 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.550683    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"
Jul 16 22:19:54.553447 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.553413    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:19:54.553577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.553452    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:19:54.556782 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.556758    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"]
Jul 16 22:19:54.665465 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.665429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78b9c6e-da44-44f9-92d3-2b5ed0ef3764-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-9t48x\" (UID: \"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"
Jul 16 22:19:54.766669 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.766636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78b9c6e-da44-44f9-92d3-2b5ed0ef3764-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-9t48x\" (UID: \"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"
Jul 16 22:19:54.767267 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.767250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78b9c6e-da44-44f9-92d3-2b5ed0ef3764-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-9t48x\" (UID: \"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"
Jul 16 22:19:54.862101 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.862066    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"
Jul 16 22:19:54.992620 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:54.992532    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"]
Jul 16 22:19:55.367658 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:55.367619    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" event={"ID":"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764","Type":"ContainerStarted","Data":"ffa9c78a7f0fe9415ac849c1524995203fb7f4d39a1c4dbca732c02a4701c42e"}
Jul 16 22:19:55.367862 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:55.367670    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" event={"ID":"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764","Type":"ContainerStarted","Data":"fede086e7b5be4ef642531b6978c5ac3b210b451c275ad60d69edd7d8ae57e09"}
Jul 16 22:19:55.384296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:55.384241    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" podStartSLOduration=1.384224342 podStartE2EDuration="1.384224342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:55.384119653 +0000 UTC m=+1951.377283813" watchObservedRunningTime="2026-07-16 22:19:55.384224342 +0000 UTC m=+1951.377388494"
Jul 16 22:19:56.324341 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.324303    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"]
Jul 16 22:19:56.327512 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.327490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"
Jul 16 22:19:56.347401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.347333    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"]
Jul 16 22:19:56.374885 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.374852    2555 generic.go:356] "Generic (PLEG): container finished" podID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerID="ffa9c78a7f0fe9415ac849c1524995203fb7f4d39a1c4dbca732c02a4701c42e" exitCode=6
Jul 16 22:19:56.375063 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.374937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" event={"ID":"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764","Type":"ContainerDied","Data":"ffa9c78a7f0fe9415ac849c1524995203fb7f4d39a1c4dbca732c02a4701c42e"}
Jul 16 22:19:56.375117 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.375096    2555 scope.go:117] "RemoveContainer" containerID="ffa9c78a7f0fe9415ac849c1524995203fb7f4d39a1c4dbca732c02a4701c42e"
Jul 16 22:19:56.377267 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.377221    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f448937-bf0b-4a22-b91b-9e8158713480-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v\" (UID: \"6f448937-bf0b-4a22-b91b-9e8158713480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"
Jul 16 22:19:56.477930 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.477895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f448937-bf0b-4a22-b91b-9e8158713480-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v\" (UID: \"6f448937-bf0b-4a22-b91b-9e8158713480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"
Jul 16 22:19:56.478498 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.478472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f448937-bf0b-4a22-b91b-9e8158713480-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v\" (UID: \"6f448937-bf0b-4a22-b91b-9e8158713480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"
Jul 16 22:19:56.637111 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.637067    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"
Jul 16 22:19:56.767375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:56.767346    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"]
Jul 16 22:19:56.768760 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:19:56.768726    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f448937_bf0b_4a22_b91b_9e8158713480.slice/crio-9f815b123a898769b5c2072f2e394379cd297cdbea89c92a3aa51e60917062b4 WatchSource:0}: Error finding container 9f815b123a898769b5c2072f2e394379cd297cdbea89c92a3aa51e60917062b4: Status 404 returned error can't find the container with id 9f815b123a898769b5c2072f2e394379cd297cdbea89c92a3aa51e60917062b4
Jul 16 22:19:57.381511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:57.381478    2555 generic.go:356] "Generic (PLEG): container finished" podID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerID="b26e5bac684822fdf80b2069d25c656b74f7e2a8c86bb11f12d6f98e4a48ccc4" exitCode=6
Jul 16 22:19:57.382020 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:57.381559    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" event={"ID":"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764","Type":"ContainerDied","Data":"b26e5bac684822fdf80b2069d25c656b74f7e2a8c86bb11f12d6f98e4a48ccc4"}
Jul 16 22:19:57.382020 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:57.381626    2555 scope.go:117] "RemoveContainer" containerID="ffa9c78a7f0fe9415ac849c1524995203fb7f4d39a1c4dbca732c02a4701c42e"
Jul 16 22:19:57.382020 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:57.381783    2555 scope.go:117] "RemoveContainer" containerID="b26e5bac684822fdf80b2069d25c656b74f7e2a8c86bb11f12d6f98e4a48ccc4"
Jul 16 22:19:57.382199 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:57.382020    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-9t48x_odh-ai-gateway-infra(e78b9c6e-da44-44f9-92d3-2b5ed0ef3764)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764"
Jul 16 22:19:57.383140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:57.383115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" event={"ID":"6f448937-bf0b-4a22-b91b-9e8158713480","Type":"ContainerStarted","Data":"3b3df1e0328e0d124acc561dac237664e490a5bef2661e51ae5b56bfb05e07d5"}
Jul 16 22:19:57.383238 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:57.383160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" event={"ID":"6f448937-bf0b-4a22-b91b-9e8158713480","Type":"ContainerStarted","Data":"9f815b123a898769b5c2072f2e394379cd297cdbea89c92a3aa51e60917062b4"}
Jul 16 22:19:58.388975 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:58.388922    2555 scope.go:117] "RemoveContainer" containerID="b26e5bac684822fdf80b2069d25c656b74f7e2a8c86bb11f12d6f98e4a48ccc4"
Jul 16 22:19:58.389408 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:58.389152    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-9t48x_odh-ai-gateway-infra(e78b9c6e-da44-44f9-92d3-2b5ed0ef3764)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764"
Jul 16 22:19:58.390176 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:58.390152    2555 generic.go:356] "Generic (PLEG): container finished" podID="6f448937-bf0b-4a22-b91b-9e8158713480" containerID="3b3df1e0328e0d124acc561dac237664e490a5bef2661e51ae5b56bfb05e07d5" exitCode=6
Jul 16 22:19:58.390314 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:58.390222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" event={"ID":"6f448937-bf0b-4a22-b91b-9e8158713480","Type":"ContainerDied","Data":"3b3df1e0328e0d124acc561dac237664e490a5bef2661e51ae5b56bfb05e07d5"}
Jul 16 22:19:58.390438 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:58.390420    2555 scope.go:117] "RemoveContainer" containerID="3b3df1e0328e0d124acc561dac237664e490a5bef2661e51ae5b56bfb05e07d5"
Jul 16 22:19:58.403924 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:58.403857    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" podStartSLOduration=2.403810267 podStartE2EDuration="2.403810267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:57.413516321 +0000 UTC m=+1953.406680485" watchObservedRunningTime="2026-07-16 22:19:58.403810267 +0000 UTC m=+1954.396974420"
Jul 16 22:19:59.396157 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:59.396123    2555 generic.go:356] "Generic (PLEG): container finished" podID="6f448937-bf0b-4a22-b91b-9e8158713480" containerID="9ca6ae6532ca1068c380c6dddf2bac51e3a5168feecc14aa67dbfcc6ca81de4a" exitCode=6
Jul 16 22:19:59.396619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:59.396184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" event={"ID":"6f448937-bf0b-4a22-b91b-9e8158713480","Type":"ContainerDied","Data":"9ca6ae6532ca1068c380c6dddf2bac51e3a5168feecc14aa67dbfcc6ca81de4a"}
Jul 16 22:19:59.396619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:59.396259    2555 scope.go:117] "RemoveContainer" containerID="3b3df1e0328e0d124acc561dac237664e490a5bef2661e51ae5b56bfb05e07d5"
Jul 16 22:19:59.396619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:19:59.396451    2555 scope.go:117] "RemoveContainer" containerID="9ca6ae6532ca1068c380c6dddf2bac51e3a5168feecc14aa67dbfcc6ca81de4a"
Jul 16 22:19:59.396758 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:19:59.396675    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v_odh-ai-gateway-infra(6f448937-bf0b-4a22-b91b-9e8158713480)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" podUID="6f448937-bf0b-4a22-b91b-9e8158713480"
Jul 16 22:20:00.402132 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:00.402103    2555 scope.go:117] "RemoveContainer" containerID="9ca6ae6532ca1068c380c6dddf2bac51e3a5168feecc14aa67dbfcc6ca81de4a"
Jul 16 22:20:00.402528 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:20:00.402304    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v_odh-ai-gateway-infra(6f448937-bf0b-4a22-b91b-9e8158713480)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" podUID="6f448937-bf0b-4a22-b91b-9e8158713480"
Jul 16 22:20:13.489860 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:13.489819    2555 scope.go:117] "RemoveContainer" containerID="b26e5bac684822fdf80b2069d25c656b74f7e2a8c86bb11f12d6f98e4a48ccc4"
Jul 16 22:20:14.462355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:14.462323    2555 generic.go:356] "Generic (PLEG): container finished" podID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerID="130cc4736692da554acc18599adadcf4d0baa48d164fda2388d3eefd580970ac" exitCode=6
Jul 16 22:20:14.462529 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:14.462406    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" event={"ID":"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764","Type":"ContainerDied","Data":"130cc4736692da554acc18599adadcf4d0baa48d164fda2388d3eefd580970ac"}
Jul 16 22:20:14.462529 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:14.462469    2555 scope.go:117] "RemoveContainer" containerID="b26e5bac684822fdf80b2069d25c656b74f7e2a8c86bb11f12d6f98e4a48ccc4"
Jul 16 22:20:14.462824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:14.462799    2555 scope.go:117] "RemoveContainer" containerID="130cc4736692da554acc18599adadcf4d0baa48d164fda2388d3eefd580970ac"
Jul 16 22:20:14.463084 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:20:14.463065    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-9t48x_odh-ai-gateway-infra(e78b9c6e-da44-44f9-92d3-2b5ed0ef3764)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764"
Jul 16 22:20:14.495543 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:14.495518    2555 scope.go:117] "RemoveContainer" containerID="9ca6ae6532ca1068c380c6dddf2bac51e3a5168feecc14aa67dbfcc6ca81de4a"
Jul 16 22:20:15.468368 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.468283    2555 generic.go:356] "Generic (PLEG): container finished" podID="6f448937-bf0b-4a22-b91b-9e8158713480" containerID="d5b51e32c4013d6fd65495837e94618f0f9be785498cde92beac976b94523b6d" exitCode=6
Jul 16 22:20:15.468515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.468361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" event={"ID":"6f448937-bf0b-4a22-b91b-9e8158713480","Type":"ContainerDied","Data":"d5b51e32c4013d6fd65495837e94618f0f9be785498cde92beac976b94523b6d"}
Jul 16 22:20:15.468515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.468420    2555 scope.go:117] "RemoveContainer" containerID="9ca6ae6532ca1068c380c6dddf2bac51e3a5168feecc14aa67dbfcc6ca81de4a"
Jul 16 22:20:15.468716 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.468687    2555 scope.go:117] "RemoveContainer" containerID="d5b51e32c4013d6fd65495837e94618f0f9be785498cde92beac976b94523b6d"
Jul 16 22:20:15.468925 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:20:15.468906    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v_odh-ai-gateway-infra(6f448937-bf0b-4a22-b91b-9e8158713480)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" podUID="6f448937-bf0b-4a22-b91b-9e8158713480"
Jul 16 22:20:15.490084 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.490061    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"]
Jul 16 22:20:15.622638 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.622616    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"
Jul 16 22:20:15.754746 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.754653    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78b9c6e-da44-44f9-92d3-2b5ed0ef3764-maas-api-service-ca\") pod \"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764\" (UID: \"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764\") "
Jul 16 22:20:15.755079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.755050    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78b9c6e-da44-44f9-92d3-2b5ed0ef3764-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" (UID: "e78b9c6e-da44-44f9-92d3-2b5ed0ef3764"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:15.855260 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:15.855219    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78b9c6e-da44-44f9-92d3-2b5ed0ef3764-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:20:16.478298 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.478262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x" event={"ID":"e78b9c6e-da44-44f9-92d3-2b5ed0ef3764","Type":"ContainerDied","Data":"fede086e7b5be4ef642531b6978c5ac3b210b451c275ad60d69edd7d8ae57e09"}
Jul 16 22:20:16.478298 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.478285    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"
Jul 16 22:20:16.478546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.478349    2555 scope.go:117] "RemoveContainer" containerID="130cc4736692da554acc18599adadcf4d0baa48d164fda2388d3eefd580970ac"
Jul 16 22:20:16.495006 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.494982    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"]
Jul 16 22:20:16.500908 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.500884    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"]
Jul 16 22:20:16.507204 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.507174    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9t48x"]
Jul 16 22:20:16.628569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.628542    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"
Jul 16 22:20:16.763779 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.763690    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f448937-bf0b-4a22-b91b-9e8158713480-maas-api-service-ca\") pod \"6f448937-bf0b-4a22-b91b-9e8158713480\" (UID: \"6f448937-bf0b-4a22-b91b-9e8158713480\") "
Jul 16 22:20:16.764062 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.764039    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f448937-bf0b-4a22-b91b-9e8158713480-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f448937-bf0b-4a22-b91b-9e8158713480" (UID: "6f448937-bf0b-4a22-b91b-9e8158713480"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:16.864435 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:16.864398    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f448937-bf0b-4a22-b91b-9e8158713480-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:20:17.484645 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:17.484608    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v" event={"ID":"6f448937-bf0b-4a22-b91b-9e8158713480","Type":"ContainerDied","Data":"9f815b123a898769b5c2072f2e394379cd297cdbea89c92a3aa51e60917062b4"}
Jul 16 22:20:17.484828 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:17.484667    2555 scope.go:117] "RemoveContainer" containerID="d5b51e32c4013d6fd65495837e94618f0f9be785498cde92beac976b94523b6d"
Jul 16 22:20:17.484828 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:17.484617    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"
Jul 16 22:20:17.507638 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:17.507610    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"]
Jul 16 22:20:17.514863 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:17.514835    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xpl7v"]
Jul 16 22:20:18.494818 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:18.494776    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" path="/var/lib/kubelet/pods/6f448937-bf0b-4a22-b91b-9e8158713480/volumes"
Jul 16 22:20:18.495325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:18.495300    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" path="/var/lib/kubelet/pods/e78b9c6e-da44-44f9-92d3-2b5ed0ef3764/volumes"
Jul 16 22:20:54.603548 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603515    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"]
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603802    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603813    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603823    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603829    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603839    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603845    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603853    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603858    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603867    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603873    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603880    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603885    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603973    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603980    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.603983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603986    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:54.604445 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.603995    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:54.607157 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.607138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"
Jul 16 22:20:54.609894 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.609877    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:20:54.610028 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.609918    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:20:54.615415 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.615390    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"]
Jul 16 22:20:54.690981 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.690933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892d80ef-710e-43fd-9956-ebbcb6efd73c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4\" (UID: \"892d80ef-710e-43fd-9956-ebbcb6efd73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"
Jul 16 22:20:54.791839 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.791796    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892d80ef-710e-43fd-9956-ebbcb6efd73c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4\" (UID: \"892d80ef-710e-43fd-9956-ebbcb6efd73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"
Jul 16 22:20:54.792447 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.792426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892d80ef-710e-43fd-9956-ebbcb6efd73c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4\" (UID: \"892d80ef-710e-43fd-9956-ebbcb6efd73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"
Jul 16 22:20:54.918110 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:54.918034    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"
Jul 16 22:20:55.065580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:55.065550    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"]
Jul 16 22:20:55.067921 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:20:55.067888    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892d80ef_710e_43fd_9956_ebbcb6efd73c.slice/crio-d411cfbd9f76f92c63c8c7235929dee0b116e574d09c670332c0e9b7cc26ddf9 WatchSource:0}: Error finding container d411cfbd9f76f92c63c8c7235929dee0b116e574d09c670332c0e9b7cc26ddf9: Status 404 returned error can't find the container with id d411cfbd9f76f92c63c8c7235929dee0b116e574d09c670332c0e9b7cc26ddf9
Jul 16 22:20:55.638007 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:55.637946    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" event={"ID":"892d80ef-710e-43fd-9956-ebbcb6efd73c","Type":"ContainerStarted","Data":"a3f5d20f208e1ebe9b99ad20e6e77c1ce043b60685ac7def53eedbf6df5e5b5f"}
Jul 16 22:20:55.638449 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:55.638020    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" event={"ID":"892d80ef-710e-43fd-9956-ebbcb6efd73c","Type":"ContainerStarted","Data":"d411cfbd9f76f92c63c8c7235929dee0b116e574d09c670332c0e9b7cc26ddf9"}
Jul 16 22:20:55.658899 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:55.658860    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" podStartSLOduration=1.6588435000000001 podStartE2EDuration="1.6588435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:55.656581504 +0000 UTC m=+2011.649745656" watchObservedRunningTime="2026-07-16 22:20:55.6588435 +0000 UTC m=+2011.652007651"
Jul 16 22:20:56.372992 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.372947    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"]
Jul 16 22:20:56.373306 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.373295    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78b9c6e-da44-44f9-92d3-2b5ed0ef3764" containerName="revoke-keys"
Jul 16 22:20:56.373350 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.373311    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f448937-bf0b-4a22-b91b-9e8158713480" containerName="revoke-keys"
Jul 16 22:20:56.376143 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.376125    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"
Jul 16 22:20:56.383778 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.383752    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"]
Jul 16 22:20:56.511984 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.511922    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/522fdc00-ff0a-436d-b13c-9128976bddd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-77767\" (UID: \"522fdc00-ff0a-436d-b13c-9128976bddd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"
Jul 16 22:20:56.612821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.612781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/522fdc00-ff0a-436d-b13c-9128976bddd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-77767\" (UID: \"522fdc00-ff0a-436d-b13c-9128976bddd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"
Jul 16 22:20:56.613440 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.613417    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/522fdc00-ff0a-436d-b13c-9128976bddd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-77767\" (UID: \"522fdc00-ff0a-436d-b13c-9128976bddd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"
Jul 16 22:20:56.643686 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.643606    2555 generic.go:356] "Generic (PLEG): container finished" podID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerID="a3f5d20f208e1ebe9b99ad20e6e77c1ce043b60685ac7def53eedbf6df5e5b5f" exitCode=6
Jul 16 22:20:56.643686 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.643672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" event={"ID":"892d80ef-710e-43fd-9956-ebbcb6efd73c","Type":"ContainerDied","Data":"a3f5d20f208e1ebe9b99ad20e6e77c1ce043b60685ac7def53eedbf6df5e5b5f"}
Jul 16 22:20:56.644142 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.643834    2555 scope.go:117] "RemoveContainer" containerID="a3f5d20f208e1ebe9b99ad20e6e77c1ce043b60685ac7def53eedbf6df5e5b5f"
Jul 16 22:20:56.687116 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.687086    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"
Jul 16 22:20:56.814722 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:56.814607    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"]
Jul 16 22:20:56.816882 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:20:56.816853    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522fdc00_ff0a_436d_b13c_9128976bddd6.slice/crio-24cb890b895a018b480a8bb228f2b308bc32a656b08610fbb18a27d5905e4c80 WatchSource:0}: Error finding container 24cb890b895a018b480a8bb228f2b308bc32a656b08610fbb18a27d5905e4c80: Status 404 returned error can't find the container with id 24cb890b895a018b480a8bb228f2b308bc32a656b08610fbb18a27d5905e4c80
Jul 16 22:20:57.649038 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.649004    2555 generic.go:356] "Generic (PLEG): container finished" podID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerID="ae933c0d9d33e11f36d91ee0495c528e549c55505b2890e723460cdece404f4e" exitCode=6
Jul 16 22:20:57.649466 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.649093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" event={"ID":"522fdc00-ff0a-436d-b13c-9128976bddd6","Type":"ContainerDied","Data":"ae933c0d9d33e11f36d91ee0495c528e549c55505b2890e723460cdece404f4e"}
Jul 16 22:20:57.649466 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.649149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" event={"ID":"522fdc00-ff0a-436d-b13c-9128976bddd6","Type":"ContainerStarted","Data":"24cb890b895a018b480a8bb228f2b308bc32a656b08610fbb18a27d5905e4c80"}
Jul 16 22:20:57.649466 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.649302    2555 scope.go:117] "RemoveContainer" containerID="ae933c0d9d33e11f36d91ee0495c528e549c55505b2890e723460cdece404f4e"
Jul 16 22:20:57.650917 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.650894    2555 generic.go:356] "Generic (PLEG): container finished" podID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerID="28d0877a5485bbdf944ce8a6adb06b2d369073cfa8fd566403a6d2df287f3abf" exitCode=6
Jul 16 22:20:57.651054 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.650934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" event={"ID":"892d80ef-710e-43fd-9956-ebbcb6efd73c","Type":"ContainerDied","Data":"28d0877a5485bbdf944ce8a6adb06b2d369073cfa8fd566403a6d2df287f3abf"}
Jul 16 22:20:57.651054 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.651001    2555 scope.go:117] "RemoveContainer" containerID="a3f5d20f208e1ebe9b99ad20e6e77c1ce043b60685ac7def53eedbf6df5e5b5f"
Jul 16 22:20:57.651140 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:57.651124    2555 scope.go:117] "RemoveContainer" containerID="28d0877a5485bbdf944ce8a6adb06b2d369073cfa8fd566403a6d2df287f3abf"
Jul 16 22:20:57.651323 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:20:57.651303    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4_odh-ai-gateway-infra(892d80ef-710e-43fd-9956-ebbcb6efd73c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c"
Jul 16 22:20:58.656922 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:58.656835    2555 generic.go:356] "Generic (PLEG): container finished" podID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerID="821559872b89a3ee082d251b411492697c4170330547417c0408d0652fbcc6e7" exitCode=6
Jul 16 22:20:58.657499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:58.656924    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" event={"ID":"522fdc00-ff0a-436d-b13c-9128976bddd6","Type":"ContainerDied","Data":"821559872b89a3ee082d251b411492697c4170330547417c0408d0652fbcc6e7"}
Jul 16 22:20:58.657499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:58.657010    2555 scope.go:117] "RemoveContainer" containerID="ae933c0d9d33e11f36d91ee0495c528e549c55505b2890e723460cdece404f4e"
Jul 16 22:20:58.657499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:58.657222    2555 scope.go:117] "RemoveContainer" containerID="821559872b89a3ee082d251b411492697c4170330547417c0408d0652fbcc6e7"
Jul 16 22:20:58.657499 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:20:58.657456    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-77767_odh-ai-gateway-infra(522fdc00-ff0a-436d-b13c-9128976bddd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6"
Jul 16 22:20:58.658784 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:58.658767    2555 scope.go:117] "RemoveContainer" containerID="28d0877a5485bbdf944ce8a6adb06b2d369073cfa8fd566403a6d2df287f3abf"
Jul 16 22:20:58.658945 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:20:58.658930    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4_odh-ai-gateway-infra(892d80ef-710e-43fd-9956-ebbcb6efd73c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c"
Jul 16 22:20:59.664707 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:20:59.664682    2555 scope.go:117] "RemoveContainer" containerID="821559872b89a3ee082d251b411492697c4170330547417c0408d0652fbcc6e7"
Jul 16 22:20:59.665124 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:20:59.664863    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-77767_odh-ai-gateway-infra(522fdc00-ff0a-436d-b13c-9128976bddd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6"
Jul 16 22:21:10.490475 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:10.490437    2555 scope.go:117] "RemoveContainer" containerID="821559872b89a3ee082d251b411492697c4170330547417c0408d0652fbcc6e7"
Jul 16 22:21:10.710579 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:10.710543    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" event={"ID":"522fdc00-ff0a-436d-b13c-9128976bddd6","Type":"ContainerStarted","Data":"eb7f3a73d987f71d97740b02a4c04711432550fa2d3c8d84bea5b3388d8f53bc"}
Jul 16 22:21:10.732383 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:10.732325    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" podStartSLOduration=14.732308462 podStartE2EDuration="14.732308462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:10.729815173 +0000 UTC m=+2026.722979326" watchObservedRunningTime="2026-07-16 22:21:10.732308462 +0000 UTC m=+2026.725472628"
Jul 16 22:21:11.717013 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:11.716979    2555 generic.go:356] "Generic (PLEG): container finished" podID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerID="eb7f3a73d987f71d97740b02a4c04711432550fa2d3c8d84bea5b3388d8f53bc" exitCode=6
Jul 16 22:21:11.717013 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:11.717000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" event={"ID":"522fdc00-ff0a-436d-b13c-9128976bddd6","Type":"ContainerDied","Data":"eb7f3a73d987f71d97740b02a4c04711432550fa2d3c8d84bea5b3388d8f53bc"}
Jul 16 22:21:11.717538 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:11.717072    2555 scope.go:117] "RemoveContainer" containerID="821559872b89a3ee082d251b411492697c4170330547417c0408d0652fbcc6e7"
Jul 16 22:21:11.717538 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:11.717373    2555 scope.go:117] "RemoveContainer" containerID="eb7f3a73d987f71d97740b02a4c04711432550fa2d3c8d84bea5b3388d8f53bc"
Jul 16 22:21:11.717634 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:11.717566    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-77767_odh-ai-gateway-infra(522fdc00-ff0a-436d-b13c-9128976bddd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6"
Jul 16 22:21:11.740604 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:11.740571    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"]
Jul 16 22:21:12.857828 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:12.857802    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"
Jul 16 22:21:12.951708 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:12.951676    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/522fdc00-ff0a-436d-b13c-9128976bddd6-maas-api-service-ca\") pod \"522fdc00-ff0a-436d-b13c-9128976bddd6\" (UID: \"522fdc00-ff0a-436d-b13c-9128976bddd6\") "
Jul 16 22:21:12.952110 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:12.952084    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522fdc00-ff0a-436d-b13c-9128976bddd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "522fdc00-ff0a-436d-b13c-9128976bddd6" (UID: "522fdc00-ff0a-436d-b13c-9128976bddd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:13.053049 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:13.052948    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/522fdc00-ff0a-436d-b13c-9128976bddd6-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:21:13.727981 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:13.727929    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767" event={"ID":"522fdc00-ff0a-436d-b13c-9128976bddd6","Type":"ContainerDied","Data":"24cb890b895a018b480a8bb228f2b308bc32a656b08610fbb18a27d5905e4c80"}
Jul 16 22:21:13.727981 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:13.727973    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"
Jul 16 22:21:13.728194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:13.728010    2555 scope.go:117] "RemoveContainer" containerID="eb7f3a73d987f71d97740b02a4c04711432550fa2d3c8d84bea5b3388d8f53bc"
Jul 16 22:21:13.750276 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:13.750248    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"]
Jul 16 22:21:13.756042 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:13.756020    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-77767"]
Jul 16 22:21:14.491308 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.491282    2555 scope.go:117] "RemoveContainer" containerID="28d0877a5485bbdf944ce8a6adb06b2d369073cfa8fd566403a6d2df287f3abf"
Jul 16 22:21:14.494532 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.494508    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" path="/var/lib/kubelet/pods/522fdc00-ff0a-436d-b13c-9128976bddd6/volumes"
Jul 16 22:21:14.958066 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958035    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"]
Jul 16 22:21:14.958339 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958328    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:14.958385 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958340    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:14.958385 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958360    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:14.958385 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958365    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:14.958491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958415    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:14.958491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958423    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:14.958491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.958428    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:14.961157 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.961141    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"
Jul 16 22:21:14.969996 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:14.969968    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"]
Jul 16 22:21:15.072865 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.072829    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53c0d41-3493-431b-9f10-0bb127e3d643-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl\" (UID: \"c53c0d41-3493-431b-9f10-0bb127e3d643\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"
Jul 16 22:21:15.173717 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.173678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53c0d41-3493-431b-9f10-0bb127e3d643-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl\" (UID: \"c53c0d41-3493-431b-9f10-0bb127e3d643\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"
Jul 16 22:21:15.174371 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.174352    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53c0d41-3493-431b-9f10-0bb127e3d643-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl\" (UID: \"c53c0d41-3493-431b-9f10-0bb127e3d643\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"
Jul 16 22:21:15.276011 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.275973    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"
Jul 16 22:21:15.609135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.609013    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"]
Jul 16 22:21:15.611628 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:21:15.611600    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53c0d41_3493_431b_9f10_0bb127e3d643.slice/crio-0e0f6ae968befa67232b4181a3982fc3793a3f520566d4cab2dfbda556ba3b15 WatchSource:0}: Error finding container 0e0f6ae968befa67232b4181a3982fc3793a3f520566d4cab2dfbda556ba3b15: Status 404 returned error can't find the container with id 0e0f6ae968befa67232b4181a3982fc3793a3f520566d4cab2dfbda556ba3b15
Jul 16 22:21:15.738787 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.738756    2555 generic.go:356] "Generic (PLEG): container finished" podID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerID="9fd0329b347c7322b30245069178c6393545412fffd31d9e093e3b388bbcc19c" exitCode=6
Jul 16 22:21:15.738983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.738830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" event={"ID":"892d80ef-710e-43fd-9956-ebbcb6efd73c","Type":"ContainerDied","Data":"9fd0329b347c7322b30245069178c6393545412fffd31d9e093e3b388bbcc19c"}
Jul 16 22:21:15.738983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.738898    2555 scope.go:117] "RemoveContainer" containerID="28d0877a5485bbdf944ce8a6adb06b2d369073cfa8fd566403a6d2df287f3abf"
Jul 16 22:21:15.739256 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.739233    2555 scope.go:117] "RemoveContainer" containerID="9fd0329b347c7322b30245069178c6393545412fffd31d9e093e3b388bbcc19c"
Jul 16 22:21:15.739502 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:15.739467    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4_odh-ai-gateway-infra(892d80ef-710e-43fd-9956-ebbcb6efd73c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c"
Jul 16 22:21:15.740125 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:15.740103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" event={"ID":"c53c0d41-3493-431b-9f10-0bb127e3d643","Type":"ContainerStarted","Data":"0e0f6ae968befa67232b4181a3982fc3793a3f520566d4cab2dfbda556ba3b15"}
Jul 16 22:21:16.747977 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:16.747859    2555 generic.go:356] "Generic (PLEG): container finished" podID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerID="708483c8e40dcf895883a66943b0bc47b5a97ed023da6c3f119937bdf3ac2247" exitCode=7
Jul 16 22:21:16.748442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:16.747949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" event={"ID":"c53c0d41-3493-431b-9f10-0bb127e3d643","Type":"ContainerDied","Data":"708483c8e40dcf895883a66943b0bc47b5a97ed023da6c3f119937bdf3ac2247"}
Jul 16 22:21:16.748442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:16.748179    2555 scope.go:117] "RemoveContainer" containerID="708483c8e40dcf895883a66943b0bc47b5a97ed023da6c3f119937bdf3ac2247"
Jul 16 22:21:16.768048 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:16.768021    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"]
Jul 16 22:21:16.899857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:16.899833    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"
Jul 16 22:21:16.995390 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:16.995366    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892d80ef-710e-43fd-9956-ebbcb6efd73c-maas-api-service-ca\") pod \"892d80ef-710e-43fd-9956-ebbcb6efd73c\" (UID: \"892d80ef-710e-43fd-9956-ebbcb6efd73c\") "
Jul 16 22:21:16.995750 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:16.995724    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892d80ef-710e-43fd-9956-ebbcb6efd73c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "892d80ef-710e-43fd-9956-ebbcb6efd73c" (UID: "892d80ef-710e-43fd-9956-ebbcb6efd73c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:17.095926 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.095894    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892d80ef-710e-43fd-9956-ebbcb6efd73c-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:21:17.754149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.754057    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"
Jul 16 22:21:17.754149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.754063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4" event={"ID":"892d80ef-710e-43fd-9956-ebbcb6efd73c","Type":"ContainerDied","Data":"d411cfbd9f76f92c63c8c7235929dee0b116e574d09c670332c0e9b7cc26ddf9"}
Jul 16 22:21:17.754688 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.754182    2555 scope.go:117] "RemoveContainer" containerID="9fd0329b347c7322b30245069178c6393545412fffd31d9e093e3b388bbcc19c"
Jul 16 22:21:17.756565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.756544    2555 generic.go:356] "Generic (PLEG): container finished" podID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerID="43333399b88e2ab05a1db38fc6e9103964264e98c17847d6dcb0c70be9fa6ff0" exitCode=7
Jul 16 22:21:17.756685 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.756592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" event={"ID":"c53c0d41-3493-431b-9f10-0bb127e3d643","Type":"ContainerDied","Data":"43333399b88e2ab05a1db38fc6e9103964264e98c17847d6dcb0c70be9fa6ff0"}
Jul 16 22:21:17.756926 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.756910    2555 scope.go:117] "RemoveContainer" containerID="43333399b88e2ab05a1db38fc6e9103964264e98c17847d6dcb0c70be9fa6ff0"
Jul 16 22:21:17.757188 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:17.757164    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl_odh-ai-gateway-infra(c53c0d41-3493-431b-9f10-0bb127e3d643)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643"
Jul 16 22:21:17.763159 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.763142    2555 scope.go:117] "RemoveContainer" containerID="708483c8e40dcf895883a66943b0bc47b5a97ed023da6c3f119937bdf3ac2247"
Jul 16 22:21:17.794972 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.794928    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"]
Jul 16 22:21:17.799296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:17.799275    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldbl4"]
Jul 16 22:21:18.494852 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:18.494810    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" path="/var/lib/kubelet/pods/892d80ef-710e-43fd-9956-ebbcb6efd73c/volumes"
Jul 16 22:21:18.768806 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:18.768722    2555 scope.go:117] "RemoveContainer" containerID="43333399b88e2ab05a1db38fc6e9103964264e98c17847d6dcb0c70be9fa6ff0"
Jul 16 22:21:18.769184 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:18.768888    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl_odh-ai-gateway-infra(c53c0d41-3493-431b-9f10-0bb127e3d643)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643"
Jul 16 22:21:32.489974 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:32.489916    2555 scope.go:117] "RemoveContainer" containerID="43333399b88e2ab05a1db38fc6e9103964264e98c17847d6dcb0c70be9fa6ff0"
Jul 16 22:21:32.835579 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:32.835540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" event={"ID":"c53c0d41-3493-431b-9f10-0bb127e3d643","Type":"ContainerStarted","Data":"44487e0ad67cf2f1b33188eb263fd75a2b7cf644b81c7e15aa48705374c96fd0"}
Jul 16 22:21:32.854924 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:32.854866    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" podStartSLOduration=18.854847456 podStartE2EDuration="18.854847456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:32.852810729 +0000 UTC m=+2048.845974880" watchObservedRunningTime="2026-07-16 22:21:32.854847456 +0000 UTC m=+2048.848011609"
Jul 16 22:21:33.842403 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:33.842368    2555 generic.go:356] "Generic (PLEG): container finished" podID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerID="44487e0ad67cf2f1b33188eb263fd75a2b7cf644b81c7e15aa48705374c96fd0" exitCode=7
Jul 16 22:21:33.842808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:33.842423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" event={"ID":"c53c0d41-3493-431b-9f10-0bb127e3d643","Type":"ContainerDied","Data":"44487e0ad67cf2f1b33188eb263fd75a2b7cf644b81c7e15aa48705374c96fd0"}
Jul 16 22:21:33.842808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:33.842472    2555 scope.go:117] "RemoveContainer" containerID="43333399b88e2ab05a1db38fc6e9103964264e98c17847d6dcb0c70be9fa6ff0"
Jul 16 22:21:33.842808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:33.842752    2555 scope.go:117] "RemoveContainer" containerID="44487e0ad67cf2f1b33188eb263fd75a2b7cf644b81c7e15aa48705374c96fd0"
Jul 16 22:21:33.842949 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:33.842936    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl_odh-ai-gateway-infra(c53c0d41-3493-431b-9f10-0bb127e3d643)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643"
Jul 16 22:21:33.865173 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:33.865132    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"]
Jul 16 22:21:34.979875 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:34.979849    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"
Jul 16 22:21:35.038423 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.038387    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53c0d41-3493-431b-9f10-0bb127e3d643-maas-api-service-ca\") pod \"c53c0d41-3493-431b-9f10-0bb127e3d643\" (UID: \"c53c0d41-3493-431b-9f10-0bb127e3d643\") "
Jul 16 22:21:35.038741 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.038718    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53c0d41-3493-431b-9f10-0bb127e3d643-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c53c0d41-3493-431b-9f10-0bb127e3d643" (UID: "c53c0d41-3493-431b-9f10-0bb127e3d643"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:35.138986 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.138870    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53c0d41-3493-431b-9f10-0bb127e3d643-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:21:35.852455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.852416    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl" event={"ID":"c53c0d41-3493-431b-9f10-0bb127e3d643","Type":"ContainerDied","Data":"0e0f6ae968befa67232b4181a3982fc3793a3f520566d4cab2dfbda556ba3b15"}
Jul 16 22:21:35.852455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.852437    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"
Jul 16 22:21:35.852699 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.852488    2555 scope.go:117] "RemoveContainer" containerID="44487e0ad67cf2f1b33188eb263fd75a2b7cf644b81c7e15aa48705374c96fd0"
Jul 16 22:21:35.878831 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.878805    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"]
Jul 16 22:21:35.887326 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:35.887301    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-cx5wl"]
Jul 16 22:21:36.494098 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:36.494068    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" path="/var/lib/kubelet/pods/c53c0d41-3493-431b-9f10-0bb127e3d643/volumes"
Jul 16 22:21:46.417058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.416975    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"]
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417400    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417417    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417440    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417448    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417463    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417471    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417480    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417488    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417505    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417512    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="522fdc00-ff0a-436d-b13c-9128976bddd6" containerName="revoke-keys"
Jul 16 22:21:46.417521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417524    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.418089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417533    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.418089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417611    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.418089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417624    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.418089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417634    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53c0d41-3493-431b-9f10-0bb127e3d643" containerName="revoke-keys"
Jul 16 22:21:46.418089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417644    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:46.418089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417654    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:46.418089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.417661    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:46.422232 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.422212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"
Jul 16 22:21:46.425331 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.425306    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:21:46.425331 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.425323    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:21:46.426190 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.426163    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"]
Jul 16 22:21:46.538636 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.538597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d60e656-7c91-4f61-a0ae-e44e8a4eb96c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz\" (UID: \"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"
Jul 16 22:21:46.639414 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.639376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d60e656-7c91-4f61-a0ae-e44e8a4eb96c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz\" (UID: \"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"
Jul 16 22:21:46.640065 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.640045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d60e656-7c91-4f61-a0ae-e44e8a4eb96c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz\" (UID: \"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"
Jul 16 22:21:46.735232 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.735141    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"
Jul 16 22:21:46.867715 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.867686    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"]
Jul 16 22:21:46.869883 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:21:46.869850    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d60e656_7c91_4f61_a0ae_e44e8a4eb96c.slice/crio-0825eaa33bdb7310ecb2fe9973035e85904ee6609cf4952a1d84f438c682a399 WatchSource:0}: Error finding container 0825eaa33bdb7310ecb2fe9973035e85904ee6609cf4952a1d84f438c682a399: Status 404 returned error can't find the container with id 0825eaa33bdb7310ecb2fe9973035e85904ee6609cf4952a1d84f438c682a399
Jul 16 22:21:46.898253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:46.898226    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" event={"ID":"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c","Type":"ContainerStarted","Data":"0825eaa33bdb7310ecb2fe9973035e85904ee6609cf4952a1d84f438c682a399"}
Jul 16 22:21:47.903195 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:47.903160    2555 generic.go:356] "Generic (PLEG): container finished" podID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerID="da3bed01a33327be5c2ea959df321acfedf02f9f3d26346971bb7968f0701bd3" exitCode=6
Jul 16 22:21:47.903644 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:47.903228    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" event={"ID":"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c","Type":"ContainerDied","Data":"da3bed01a33327be5c2ea959df321acfedf02f9f3d26346971bb7968f0701bd3"}
Jul 16 22:21:47.903644 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:47.903446    2555 scope.go:117] "RemoveContainer" containerID="da3bed01a33327be5c2ea959df321acfedf02f9f3d26346971bb7968f0701bd3"
Jul 16 22:21:48.909409 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:48.909371    2555 generic.go:356] "Generic (PLEG): container finished" podID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerID="d3b65a8ab441e9f3269fa7b0d89e957e2337820ce620809baa2bebb6addd48f8" exitCode=6
Jul 16 22:21:48.909821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:48.909434    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" event={"ID":"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c","Type":"ContainerDied","Data":"d3b65a8ab441e9f3269fa7b0d89e957e2337820ce620809baa2bebb6addd48f8"}
Jul 16 22:21:48.909821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:48.909493    2555 scope.go:117] "RemoveContainer" containerID="da3bed01a33327be5c2ea959df321acfedf02f9f3d26346971bb7968f0701bd3"
Jul 16 22:21:48.909821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:48.909670    2555 scope.go:117] "RemoveContainer" containerID="d3b65a8ab441e9f3269fa7b0d89e957e2337820ce620809baa2bebb6addd48f8"
Jul 16 22:21:48.909981 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:48.909923    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz_odh-ai-gateway-infra(9d60e656-7c91-4f61-a0ae-e44e8a4eb96c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c"
Jul 16 22:21:49.915272 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:49.915242    2555 scope.go:117] "RemoveContainer" containerID="d3b65a8ab441e9f3269fa7b0d89e957e2337820ce620809baa2bebb6addd48f8"
Jul 16 22:21:49.915647 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:49.915445    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz_odh-ai-gateway-infra(9d60e656-7c91-4f61-a0ae-e44e8a4eb96c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c"
Jul 16 22:21:54.656733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.656695    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"]
Jul 16 22:21:54.657162 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.657051    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:54.657162 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.657064    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="892d80ef-710e-43fd-9956-ebbcb6efd73c" containerName="revoke-keys"
Jul 16 22:21:54.661126 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.661105    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"
Jul 16 22:21:54.669009 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.668981    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"]
Jul 16 22:21:54.810321 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.810287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed05b49c-827a-4b7f-832e-3731ff99bf72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5grvk\" (UID: \"ed05b49c-827a-4b7f-832e-3731ff99bf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"
Jul 16 22:21:54.911677 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.911589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed05b49c-827a-4b7f-832e-3731ff99bf72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5grvk\" (UID: \"ed05b49c-827a-4b7f-832e-3731ff99bf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"
Jul 16 22:21:54.912226 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.912210    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed05b49c-827a-4b7f-832e-3731ff99bf72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5grvk\" (UID: \"ed05b49c-827a-4b7f-832e-3731ff99bf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"
Jul 16 22:21:54.972428 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:54.972377    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"
Jul 16 22:21:55.114653 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:55.114620    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"]
Jul 16 22:21:55.117447 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:21:55.117412    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded05b49c_827a_4b7f_832e_3731ff99bf72.slice/crio-3f1eebc27bc30abd2484314261ba56996fd9519f95082eb5307d6dad4257308f WatchSource:0}: Error finding container 3f1eebc27bc30abd2484314261ba56996fd9519f95082eb5307d6dad4257308f: Status 404 returned error can't find the container with id 3f1eebc27bc30abd2484314261ba56996fd9519f95082eb5307d6dad4257308f
Jul 16 22:21:55.942544 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:55.942513    2555 generic.go:356] "Generic (PLEG): container finished" podID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerID="18289a484eb969d0056d99c3c1d53e101424085960bbe85974641c6531bd7c99" exitCode=6
Jul 16 22:21:55.942990 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:55.942569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" event={"ID":"ed05b49c-827a-4b7f-832e-3731ff99bf72","Type":"ContainerDied","Data":"18289a484eb969d0056d99c3c1d53e101424085960bbe85974641c6531bd7c99"}
Jul 16 22:21:55.942990 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:55.942609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" event={"ID":"ed05b49c-827a-4b7f-832e-3731ff99bf72","Type":"ContainerStarted","Data":"3f1eebc27bc30abd2484314261ba56996fd9519f95082eb5307d6dad4257308f"}
Jul 16 22:21:55.942990 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:55.942784    2555 scope.go:117] "RemoveContainer" containerID="18289a484eb969d0056d99c3c1d53e101424085960bbe85974641c6531bd7c99"
Jul 16 22:21:56.948474 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:56.948437    2555 generic.go:356] "Generic (PLEG): container finished" podID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerID="93c3bf26110ba31ebbb076b9fc77457f13eb9d8e69bd2ea25d619ca22ce6a276" exitCode=6
Jul 16 22:21:56.948896 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:56.948527    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" event={"ID":"ed05b49c-827a-4b7f-832e-3731ff99bf72","Type":"ContainerDied","Data":"93c3bf26110ba31ebbb076b9fc77457f13eb9d8e69bd2ea25d619ca22ce6a276"}
Jul 16 22:21:56.948896 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:56.948590    2555 scope.go:117] "RemoveContainer" containerID="18289a484eb969d0056d99c3c1d53e101424085960bbe85974641c6531bd7c99"
Jul 16 22:21:56.948896 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:56.948849    2555 scope.go:117] "RemoveContainer" containerID="93c3bf26110ba31ebbb076b9fc77457f13eb9d8e69bd2ea25d619ca22ce6a276"
Jul 16 22:21:56.949090 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:56.949072    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5grvk_odh-ai-gateway-infra(ed05b49c-827a-4b7f-832e-3731ff99bf72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72"
Jul 16 22:21:57.954790 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:21:57.954760    2555 scope.go:117] "RemoveContainer" containerID="93c3bf26110ba31ebbb076b9fc77457f13eb9d8e69bd2ea25d619ca22ce6a276"
Jul 16 22:21:57.955211 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:21:57.954994    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5grvk_odh-ai-gateway-infra(ed05b49c-827a-4b7f-832e-3731ff99bf72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72"
Jul 16 22:22:03.490135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:03.490103    2555 scope.go:117] "RemoveContainer" containerID="d3b65a8ab441e9f3269fa7b0d89e957e2337820ce620809baa2bebb6addd48f8"
Jul 16 22:22:03.980401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:03.980365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" event={"ID":"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c","Type":"ContainerStarted","Data":"87971bca55349aa18de236441b9bb3c6ec2a637ee7fcbad41582caf08488522f"}
Jul 16 22:22:04.000484 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:04.000432    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" podStartSLOduration=18.000415592 podStartE2EDuration="18.000415592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:03.999750676 +0000 UTC m=+2079.992914828" watchObservedRunningTime="2026-07-16 22:22:04.000415592 +0000 UTC m=+2079.993579745"
Jul 16 22:22:04.986718 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:04.986684    2555 generic.go:356] "Generic (PLEG): container finished" podID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerID="87971bca55349aa18de236441b9bb3c6ec2a637ee7fcbad41582caf08488522f" exitCode=6
Jul 16 22:22:04.987253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:04.986745    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" event={"ID":"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c","Type":"ContainerDied","Data":"87971bca55349aa18de236441b9bb3c6ec2a637ee7fcbad41582caf08488522f"}
Jul 16 22:22:04.987253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:04.986792    2555 scope.go:117] "RemoveContainer" containerID="d3b65a8ab441e9f3269fa7b0d89e957e2337820ce620809baa2bebb6addd48f8"
Jul 16 22:22:04.987253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:04.987117    2555 scope.go:117] "RemoveContainer" containerID="87971bca55349aa18de236441b9bb3c6ec2a637ee7fcbad41582caf08488522f"
Jul 16 22:22:04.987423 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:04.987374    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz_odh-ai-gateway-infra(9d60e656-7c91-4f61-a0ae-e44e8a4eb96c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c"
Jul 16 22:22:05.042384 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:05.042345    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"]
Jul 16 22:22:06.120072 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:06.120043    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"
Jul 16 22:22:06.212353 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:06.212315    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d60e656-7c91-4f61-a0ae-e44e8a4eb96c-maas-api-service-ca\") pod \"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c\" (UID: \"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c\") "
Jul 16 22:22:06.212703 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:06.212682    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d60e656-7c91-4f61-a0ae-e44e8a4eb96c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" (UID: "9d60e656-7c91-4f61-a0ae-e44e8a4eb96c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:06.313897 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:06.313806    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d60e656-7c91-4f61-a0ae-e44e8a4eb96c-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.997182 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:06.997155    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"
Jul 16 22:22:06.997182 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:06.997159    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz" event={"ID":"9d60e656-7c91-4f61-a0ae-e44e8a4eb96c","Type":"ContainerDied","Data":"0825eaa33bdb7310ecb2fe9973035e85904ee6609cf4952a1d84f438c682a399"}
Jul 16 22:22:06.997446 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:06.997222    2555 scope.go:117] "RemoveContainer" containerID="87971bca55349aa18de236441b9bb3c6ec2a637ee7fcbad41582caf08488522f"
Jul 16 22:22:07.014097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:07.014071    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"]
Jul 16 22:22:07.020561 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:07.020532    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-kdbdz"]
Jul 16 22:22:08.494658 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:08.494623    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" path="/var/lib/kubelet/pods/9d60e656-7c91-4f61-a0ae-e44e8a4eb96c/volumes"
Jul 16 22:22:10.495650 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:10.495622    2555 scope.go:117] "RemoveContainer" containerID="93c3bf26110ba31ebbb076b9fc77457f13eb9d8e69bd2ea25d619ca22ce6a276"
Jul 16 22:22:11.016181 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:11.016095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" event={"ID":"ed05b49c-827a-4b7f-832e-3731ff99bf72","Type":"ContainerStarted","Data":"e9597d8c0edbae0800b79a92fa7de979e5e6dba42a7c5397699f672bb40d8395"}
Jul 16 22:22:11.045301 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:11.045258    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" podStartSLOduration=17.045242613 podStartE2EDuration="17.045242613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:11.044302977 +0000 UTC m=+2087.037467129" watchObservedRunningTime="2026-07-16 22:22:11.045242613 +0000 UTC m=+2087.038406765"
Jul 16 22:22:12.022340 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:12.022307    2555 generic.go:356] "Generic (PLEG): container finished" podID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerID="e9597d8c0edbae0800b79a92fa7de979e5e6dba42a7c5397699f672bb40d8395" exitCode=6
Jul 16 22:22:12.022782 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:12.022362    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" event={"ID":"ed05b49c-827a-4b7f-832e-3731ff99bf72","Type":"ContainerDied","Data":"e9597d8c0edbae0800b79a92fa7de979e5e6dba42a7c5397699f672bb40d8395"}
Jul 16 22:22:12.022782 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:12.022413    2555 scope.go:117] "RemoveContainer" containerID="93c3bf26110ba31ebbb076b9fc77457f13eb9d8e69bd2ea25d619ca22ce6a276"
Jul 16 22:22:12.022782 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:12.022685    2555 scope.go:117] "RemoveContainer" containerID="e9597d8c0edbae0800b79a92fa7de979e5e6dba42a7c5397699f672bb40d8395"
Jul 16 22:22:12.022969 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:12.022932    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5grvk_odh-ai-gateway-infra(ed05b49c-827a-4b7f-832e-3731ff99bf72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72"
Jul 16 22:22:12.055601 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:12.055573    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"]
Jul 16 22:22:13.159612 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:13.159584    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"
Jul 16 22:22:13.271824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:13.271784    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed05b49c-827a-4b7f-832e-3731ff99bf72-maas-api-service-ca\") pod \"ed05b49c-827a-4b7f-832e-3731ff99bf72\" (UID: \"ed05b49c-827a-4b7f-832e-3731ff99bf72\") "
Jul 16 22:22:13.272215 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:13.272193    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed05b49c-827a-4b7f-832e-3731ff99bf72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed05b49c-827a-4b7f-832e-3731ff99bf72" (UID: "ed05b49c-827a-4b7f-832e-3731ff99bf72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:13.372992 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:13.372946    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed05b49c-827a-4b7f-832e-3731ff99bf72-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:22:14.035232 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:14.035202    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"
Jul 16 22:22:14.035424 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:14.035202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk" event={"ID":"ed05b49c-827a-4b7f-832e-3731ff99bf72","Type":"ContainerDied","Data":"3f1eebc27bc30abd2484314261ba56996fd9519f95082eb5307d6dad4257308f"}
Jul 16 22:22:14.035424 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:14.035338    2555 scope.go:117] "RemoveContainer" containerID="e9597d8c0edbae0800b79a92fa7de979e5e6dba42a7c5397699f672bb40d8395"
Jul 16 22:22:14.057990 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:14.057951    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"]
Jul 16 22:22:14.061416 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:14.061390    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5grvk"]
Jul 16 22:22:14.494783 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:14.494745    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" path="/var/lib/kubelet/pods/ed05b49c-827a-4b7f-832e-3731ff99bf72/volumes"
Jul 16 22:22:15.003478 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003438    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"]
Jul 16 22:22:15.003734 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003722    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003735    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003742    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003748    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003762    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003768    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003777    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003782    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003790    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:15.003792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003795    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:15.004078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003835    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.004078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003842    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.004078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003848    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d60e656-7c91-4f61-a0ae-e44e8a4eb96c" containerName="revoke-keys"
Jul 16 22:22:15.004078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003857    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:15.004078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.003863    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:15.008046 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.008025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"
Jul 16 22:22:15.010774 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.010741    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:22:15.010774 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.010748    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:22:15.014290 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.014254    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"]
Jul 16 22:22:15.190810 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.190767    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eae6b85-0468-462d-a936-2e361fbed57d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r\" (UID: \"6eae6b85-0468-462d-a936-2e361fbed57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"
Jul 16 22:22:15.292362 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.292262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eae6b85-0468-462d-a936-2e361fbed57d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r\" (UID: \"6eae6b85-0468-462d-a936-2e361fbed57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"
Jul 16 22:22:15.292863 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.292844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eae6b85-0468-462d-a936-2e361fbed57d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r\" (UID: \"6eae6b85-0468-462d-a936-2e361fbed57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"
Jul 16 22:22:15.319789 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.319765    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"
Jul 16 22:22:15.444749 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:15.444726    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"]
Jul 16 22:22:15.446788 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:22:15.446761    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eae6b85_0468_462d_a936_2e361fbed57d.slice/crio-f5dc7e98fc81508b99f3e7e7068ad08b2795c35aa468e9b6726db6645746e0a4 WatchSource:0}: Error finding container f5dc7e98fc81508b99f3e7e7068ad08b2795c35aa468e9b6726db6645746e0a4: Status 404 returned error can't find the container with id f5dc7e98fc81508b99f3e7e7068ad08b2795c35aa468e9b6726db6645746e0a4
Jul 16 22:22:16.052619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:16.052591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" event={"ID":"6eae6b85-0468-462d-a936-2e361fbed57d","Type":"ContainerStarted","Data":"b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5"}
Jul 16 22:22:16.053001 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:16.052647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" event={"ID":"6eae6b85-0468-462d-a936-2e361fbed57d","Type":"ContainerStarted","Data":"f5dc7e98fc81508b99f3e7e7068ad08b2795c35aa468e9b6726db6645746e0a4"}
Jul 16 22:22:16.053001 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:16.052826    2555 scope.go:117] "RemoveContainer" containerID="b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5"
Jul 16 22:22:17.058462 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.058367    2555 generic.go:356] "Generic (PLEG): container finished" podID="6eae6b85-0468-462d-a936-2e361fbed57d" containerID="b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5" exitCode=7
Jul 16 22:22:17.058462 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.058393    2555 generic.go:356] "Generic (PLEG): container finished" podID="6eae6b85-0468-462d-a936-2e361fbed57d" containerID="b279142de9eeac8bba9d91291b7fef527f61457fd8ac89295a18836d9ee77b07" exitCode=7
Jul 16 22:22:17.058462 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.058443    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" event={"ID":"6eae6b85-0468-462d-a936-2e361fbed57d","Type":"ContainerDied","Data":"b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5"}
Jul 16 22:22:17.059036 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.058496    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" event={"ID":"6eae6b85-0468-462d-a936-2e361fbed57d","Type":"ContainerDied","Data":"b279142de9eeac8bba9d91291b7fef527f61457fd8ac89295a18836d9ee77b07"}
Jul 16 22:22:17.059036 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.058583    2555 scope.go:117] "RemoveContainer" containerID="b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5"
Jul 16 22:22:17.059036 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.058650    2555 scope.go:117] "RemoveContainer" containerID="b279142de9eeac8bba9d91291b7fef527f61457fd8ac89295a18836d9ee77b07"
Jul 16 22:22:17.059036 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:17.058902    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r_odh-ai-gateway-infra(6eae6b85-0468-462d-a936-2e361fbed57d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" podUID="6eae6b85-0468-462d-a936-2e361fbed57d"
Jul 16 22:22:17.068842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.068817    2555 scope.go:117] "RemoveContainer" containerID="b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5"
Jul 16 22:22:17.069119 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:17.069104    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5\": container with ID starting with b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5 not found: ID does not exist" containerID="b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5"
Jul 16 22:22:17.069197 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:17.069127    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5"} err="failed to get container status \"b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5\": rpc error: code = NotFound desc = could not find container \"b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5\": container with ID starting with b0f5bc4714b2c03be788046c140ee30526d2c0052cf35c9ede188294f10fd7d5 not found: ID does not exist"
Jul 16 22:22:18.064175 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:18.064143    2555 scope.go:117] "RemoveContainer" containerID="b279142de9eeac8bba9d91291b7fef527f61457fd8ac89295a18836d9ee77b07"
Jul 16 22:22:18.064552 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:18.064332    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r_odh-ai-gateway-infra(6eae6b85-0468-462d-a936-2e361fbed57d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" podUID="6eae6b85-0468-462d-a936-2e361fbed57d"
Jul 16 22:22:30.490317 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:30.490287    2555 scope.go:117] "RemoveContainer" containerID="b279142de9eeac8bba9d91291b7fef527f61457fd8ac89295a18836d9ee77b07"
Jul 16 22:22:31.121695 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:31.121661    2555 generic.go:356] "Generic (PLEG): container finished" podID="6eae6b85-0468-462d-a936-2e361fbed57d" containerID="1cea75ec795a01342935c81df463dfa82e8c1bbedef866c3979f135f7913ed77" exitCode=7
Jul 16 22:22:31.121892 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:31.121739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" event={"ID":"6eae6b85-0468-462d-a936-2e361fbed57d","Type":"ContainerDied","Data":"1cea75ec795a01342935c81df463dfa82e8c1bbedef866c3979f135f7913ed77"}
Jul 16 22:22:31.121892 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:31.121815    2555 scope.go:117] "RemoveContainer" containerID="b279142de9eeac8bba9d91291b7fef527f61457fd8ac89295a18836d9ee77b07"
Jul 16 22:22:31.122226 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:31.122202    2555 scope.go:117] "RemoveContainer" containerID="1cea75ec795a01342935c81df463dfa82e8c1bbedef866c3979f135f7913ed77"
Jul 16 22:22:31.122451 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:31.122432    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r_odh-ai-gateway-infra(6eae6b85-0468-462d-a936-2e361fbed57d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" podUID="6eae6b85-0468-462d-a936-2e361fbed57d"
Jul 16 22:22:32.156396 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:32.156363    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"]
Jul 16 22:22:32.283882 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:32.283852    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"
Jul 16 22:22:32.338617 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:32.338580    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eae6b85-0468-462d-a936-2e361fbed57d-maas-api-service-ca\") pod \"6eae6b85-0468-462d-a936-2e361fbed57d\" (UID: \"6eae6b85-0468-462d-a936-2e361fbed57d\") "
Jul 16 22:22:32.338940 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:32.338917    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eae6b85-0468-462d-a936-2e361fbed57d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6eae6b85-0468-462d-a936-2e361fbed57d" (UID: "6eae6b85-0468-462d-a936-2e361fbed57d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:32.439626 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:32.439543    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eae6b85-0468-462d-a936-2e361fbed57d-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:22:33.133666 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:33.133640    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"
Jul 16 22:22:33.133874 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:33.133628    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r" event={"ID":"6eae6b85-0468-462d-a936-2e361fbed57d","Type":"ContainerDied","Data":"f5dc7e98fc81508b99f3e7e7068ad08b2795c35aa468e9b6726db6645746e0a4"}
Jul 16 22:22:33.133874 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:33.133814    2555 scope.go:117] "RemoveContainer" containerID="1cea75ec795a01342935c81df463dfa82e8c1bbedef866c3979f135f7913ed77"
Jul 16 22:22:33.152513 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:33.152487    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"]
Jul 16 22:22:33.158282 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:33.158258    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9v48r"]
Jul 16 22:22:34.494314 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:34.494278    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" path="/var/lib/kubelet/pods/6eae6b85-0468-462d-a936-2e361fbed57d/volumes"
Jul 16 22:22:44.735782 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.735744    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"]
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736088    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736101    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736119    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736125    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736138    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736144    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736190    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736196    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:44.736224 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.736204    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed05b49c-827a-4b7f-832e-3731ff99bf72" containerName="revoke-keys"
Jul 16 22:22:44.740573 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.740553    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"
Jul 16 22:22:44.743806 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.743782    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:22:44.744917 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.744899    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:22:44.754426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.754405    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"]
Jul 16 22:22:44.843503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.843463    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e0a7cc-5417-4436-9ccb-2729599440bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-2htxt\" (UID: \"b3e0a7cc-5417-4436-9ccb-2729599440bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"
Jul 16 22:22:44.944343 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.944315    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e0a7cc-5417-4436-9ccb-2729599440bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-2htxt\" (UID: \"b3e0a7cc-5417-4436-9ccb-2729599440bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"
Jul 16 22:22:44.944942 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:44.944924    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e0a7cc-5417-4436-9ccb-2729599440bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-2htxt\" (UID: \"b3e0a7cc-5417-4436-9ccb-2729599440bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"
Jul 16 22:22:45.050308 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:45.050225    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"
Jul 16 22:22:45.195918 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:45.195895    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"]
Jul 16 22:22:45.198106 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:22:45.198070    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e0a7cc_5417_4436_9ccb_2729599440bf.slice/crio-f7754552750e6deacaec90df47af9a72b1d360a59def2d40e7ccc453f867f788 WatchSource:0}: Error finding container f7754552750e6deacaec90df47af9a72b1d360a59def2d40e7ccc453f867f788: Status 404 returned error can't find the container with id f7754552750e6deacaec90df47af9a72b1d360a59def2d40e7ccc453f867f788
Jul 16 22:22:46.190035 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.189999    2555 generic.go:356] "Generic (PLEG): container finished" podID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerID="4b9536477671effb58c38acd14b4fac4e9ad144642c9dcba1924c59e8f3a31c1" exitCode=6
Jul 16 22:22:46.190410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.190086    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" event={"ID":"b3e0a7cc-5417-4436-9ccb-2729599440bf","Type":"ContainerDied","Data":"4b9536477671effb58c38acd14b4fac4e9ad144642c9dcba1924c59e8f3a31c1"}
Jul 16 22:22:46.190410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.190130    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" event={"ID":"b3e0a7cc-5417-4436-9ccb-2729599440bf","Type":"ContainerStarted","Data":"f7754552750e6deacaec90df47af9a72b1d360a59def2d40e7ccc453f867f788"}
Jul 16 22:22:46.190410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.190293    2555 scope.go:117] "RemoveContainer" containerID="4b9536477671effb58c38acd14b4fac4e9ad144642c9dcba1924c59e8f3a31c1"
Jul 16 22:22:46.461480 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.461401    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"]
Jul 16 22:22:46.461767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.461754    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:46.461767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.461768    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:46.461900 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.461819    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eae6b85-0468-462d-a936-2e361fbed57d" containerName="revoke-keys"
Jul 16 22:22:46.464733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.464714    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"
Jul 16 22:22:46.472419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.472394    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"]
Jul 16 22:22:46.559548 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.559506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb4a71-36f5-49d6-8efe-378cdfb8a7e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r\" (UID: \"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"
Jul 16 22:22:46.660950 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.660908    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb4a71-36f5-49d6-8efe-378cdfb8a7e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r\" (UID: \"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"
Jul 16 22:22:46.661546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.661527    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb4a71-36f5-49d6-8efe-378cdfb8a7e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r\" (UID: \"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"
Jul 16 22:22:46.776407 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.776374    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"
Jul 16 22:22:46.915079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:46.915042    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"]
Jul 16 22:22:46.918136 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:22:46.918103    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41eb4a71_36f5_49d6_8efe_378cdfb8a7e9.slice/crio-d54de945c5015bef0a718cfaa85f12c54cd24a20d310f506b6d0290b3731448b WatchSource:0}: Error finding container d54de945c5015bef0a718cfaa85f12c54cd24a20d310f506b6d0290b3731448b: Status 404 returned error can't find the container with id d54de945c5015bef0a718cfaa85f12c54cd24a20d310f506b6d0290b3731448b
Jul 16 22:22:47.197196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:47.197148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" event={"ID":"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9","Type":"ContainerStarted","Data":"cc3266109aa76910eb3d630220c2bee6cfc033f4337e615afa2010cb85dceb97"}
Jul 16 22:22:47.197680 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:47.197226    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" event={"ID":"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9","Type":"ContainerStarted","Data":"d54de945c5015bef0a718cfaa85f12c54cd24a20d310f506b6d0290b3731448b"}
Jul 16 22:22:47.199005 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:47.198952    2555 generic.go:356] "Generic (PLEG): container finished" podID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerID="09e31663b8452b60fb6803411a28c674132f58dccfa7740c278777ee39c3c093" exitCode=6
Jul 16 22:22:47.199136 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:47.199019    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" event={"ID":"b3e0a7cc-5417-4436-9ccb-2729599440bf","Type":"ContainerDied","Data":"09e31663b8452b60fb6803411a28c674132f58dccfa7740c278777ee39c3c093"}
Jul 16 22:22:47.199136 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:47.199075    2555 scope.go:117] "RemoveContainer" containerID="4b9536477671effb58c38acd14b4fac4e9ad144642c9dcba1924c59e8f3a31c1"
Jul 16 22:22:47.199289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:47.199269    2555 scope.go:117] "RemoveContainer" containerID="09e31663b8452b60fb6803411a28c674132f58dccfa7740c278777ee39c3c093"
Jul 16 22:22:47.199517 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:47.199492    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-2htxt_odh-ai-gateway-infra(b3e0a7cc-5417-4436-9ccb-2729599440bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf"
Jul 16 22:22:47.221725 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:47.221673    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" podStartSLOduration=1.22165578 podStartE2EDuration="1.22165578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:47.22147042 +0000 UTC m=+2123.214634572" watchObservedRunningTime="2026-07-16 22:22:47.22165578 +0000 UTC m=+2123.214819932"
Jul 16 22:22:48.204677 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:48.204647    2555 scope.go:117] "RemoveContainer" containerID="09e31663b8452b60fb6803411a28c674132f58dccfa7740c278777ee39c3c093"
Jul 16 22:22:48.205134 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:48.204857    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-2htxt_odh-ai-gateway-infra(b3e0a7cc-5417-4436-9ccb-2729599440bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf"
Jul 16 22:22:48.205914 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:48.205892    2555 generic.go:356] "Generic (PLEG): container finished" podID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerID="cc3266109aa76910eb3d630220c2bee6cfc033f4337e615afa2010cb85dceb97" exitCode=6
Jul 16 22:22:48.206024 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:48.205970    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" event={"ID":"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9","Type":"ContainerDied","Data":"cc3266109aa76910eb3d630220c2bee6cfc033f4337e615afa2010cb85dceb97"}
Jul 16 22:22:48.206158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:48.206146    2555 scope.go:117] "RemoveContainer" containerID="cc3266109aa76910eb3d630220c2bee6cfc033f4337e615afa2010cb85dceb97"
Jul 16 22:22:49.211801 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:49.211769    2555 generic.go:356] "Generic (PLEG): container finished" podID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerID="594987ab59055b24b9534b51ffaffeeced68e350b9a744f304704bd54d3b208f" exitCode=6
Jul 16 22:22:49.212261 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:49.211837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" event={"ID":"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9","Type":"ContainerDied","Data":"594987ab59055b24b9534b51ffaffeeced68e350b9a744f304704bd54d3b208f"}
Jul 16 22:22:49.212261 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:49.211908    2555 scope.go:117] "RemoveContainer" containerID="cc3266109aa76910eb3d630220c2bee6cfc033f4337e615afa2010cb85dceb97"
Jul 16 22:22:49.212261 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:49.212102    2555 scope.go:117] "RemoveContainer" containerID="594987ab59055b24b9534b51ffaffeeced68e350b9a744f304704bd54d3b208f"
Jul 16 22:22:49.212381 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:49.212323    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r_odh-ai-gateway-infra(41eb4a71-36f5-49d6-8efe-378cdfb8a7e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9"
Jul 16 22:22:50.217920 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:22:50.217893    2555 scope.go:117] "RemoveContainer" containerID="594987ab59055b24b9534b51ffaffeeced68e350b9a744f304704bd54d3b208f"
Jul 16 22:22:50.218318 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:22:50.218110    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r_odh-ai-gateway-infra(41eb4a71-36f5-49d6-8efe-378cdfb8a7e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9"
Jul 16 22:23:01.489677 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:01.489648    2555 scope.go:117] "RemoveContainer" containerID="09e31663b8452b60fb6803411a28c674132f58dccfa7740c278777ee39c3c093"
Jul 16 22:23:02.267746 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:02.267711    2555 generic.go:356] "Generic (PLEG): container finished" podID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerID="215d2afa032cf159ceb2ac72c345c8f052b21bea59fa3ead5bd20716d0d754e7" exitCode=6
Jul 16 22:23:02.267946 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:02.267787    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" event={"ID":"b3e0a7cc-5417-4436-9ccb-2729599440bf","Type":"ContainerDied","Data":"215d2afa032cf159ceb2ac72c345c8f052b21bea59fa3ead5bd20716d0d754e7"}
Jul 16 22:23:02.267946 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:02.267851    2555 scope.go:117] "RemoveContainer" containerID="09e31663b8452b60fb6803411a28c674132f58dccfa7740c278777ee39c3c093"
Jul 16 22:23:02.268220 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:02.268200    2555 scope.go:117] "RemoveContainer" containerID="215d2afa032cf159ceb2ac72c345c8f052b21bea59fa3ead5bd20716d0d754e7"
Jul 16 22:23:02.268417 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:02.268398    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-2htxt_odh-ai-gateway-infra(b3e0a7cc-5417-4436-9ccb-2729599440bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf"
Jul 16 22:23:03.310271 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:03.310230    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"]
Jul 16 22:23:03.442015 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:03.441991    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"
Jul 16 22:23:03.617356 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:03.617322    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e0a7cc-5417-4436-9ccb-2729599440bf-maas-api-service-ca\") pod \"b3e0a7cc-5417-4436-9ccb-2729599440bf\" (UID: \"b3e0a7cc-5417-4436-9ccb-2729599440bf\") "
Jul 16 22:23:03.617692 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:03.617667    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e0a7cc-5417-4436-9ccb-2729599440bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3e0a7cc-5417-4436-9ccb-2729599440bf" (UID: "b3e0a7cc-5417-4436-9ccb-2729599440bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:03.718234 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:03.718196    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e0a7cc-5417-4436-9ccb-2729599440bf-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:23:04.277874 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:04.277844    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"
Jul 16 22:23:04.278073 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:04.277846    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt" event={"ID":"b3e0a7cc-5417-4436-9ccb-2729599440bf","Type":"ContainerDied","Data":"f7754552750e6deacaec90df47af9a72b1d360a59def2d40e7ccc453f867f788"}
Jul 16 22:23:04.278073 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:04.277998    2555 scope.go:117] "RemoveContainer" containerID="215d2afa032cf159ceb2ac72c345c8f052b21bea59fa3ead5bd20716d0d754e7"
Jul 16 22:23:04.312267 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:04.312234    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"]
Jul 16 22:23:04.313586 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:04.313564    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2htxt"]
Jul 16 22:23:04.491751 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:04.491724    2555 scope.go:117] "RemoveContainer" containerID="594987ab59055b24b9534b51ffaffeeced68e350b9a744f304704bd54d3b208f"
Jul 16 22:23:04.495168 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:04.495146    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" path="/var/lib/kubelet/pods/b3e0a7cc-5417-4436-9ccb-2729599440bf/volumes"
Jul 16 22:23:05.107972 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.107922    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"]
Jul 16 22:23:05.108241 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.108229    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:05.108330 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.108242    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:05.108330 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.108262    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:05.108330 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.108267    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:05.108425 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.108333    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:05.108425 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.108340    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:05.108425 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.108346    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:05.111304 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.111289    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"
Jul 16 22:23:05.128502 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.128469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"]
Jul 16 22:23:05.233088 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.233057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f30b043-2edb-400a-9a0f-e10350f67802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7\" (UID: \"8f30b043-2edb-400a-9a0f-e10350f67802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"
Jul 16 22:23:05.285125 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.285090    2555 generic.go:356] "Generic (PLEG): container finished" podID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerID="274d852b876ab5d106291b3223a9480d175af8b860b57bd0e547e5c3b93cd1ce" exitCode=6
Jul 16 22:23:05.285303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.285134    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" event={"ID":"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9","Type":"ContainerDied","Data":"274d852b876ab5d106291b3223a9480d175af8b860b57bd0e547e5c3b93cd1ce"}
Jul 16 22:23:05.285303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.285185    2555 scope.go:117] "RemoveContainer" containerID="594987ab59055b24b9534b51ffaffeeced68e350b9a744f304704bd54d3b208f"
Jul 16 22:23:05.285533 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.285504    2555 scope.go:117] "RemoveContainer" containerID="274d852b876ab5d106291b3223a9480d175af8b860b57bd0e547e5c3b93cd1ce"
Jul 16 22:23:05.285774 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:05.285755    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r_odh-ai-gateway-infra(41eb4a71-36f5-49d6-8efe-378cdfb8a7e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9"
Jul 16 22:23:05.334013 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.333947    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f30b043-2edb-400a-9a0f-e10350f67802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7\" (UID: \"8f30b043-2edb-400a-9a0f-e10350f67802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"
Jul 16 22:23:05.334577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.334558    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f30b043-2edb-400a-9a0f-e10350f67802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7\" (UID: \"8f30b043-2edb-400a-9a0f-e10350f67802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"
Jul 16 22:23:05.421698 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.421657    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"
Jul 16 22:23:05.548815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:05.548744    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"]
Jul 16 22:23:05.551433 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:23:05.551407    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f30b043_2edb_400a_9a0f_e10350f67802.slice/crio-6361ea1c4a8015455773d47f65b44428176ab5232640cc989fd96ec79863fad4 WatchSource:0}: Error finding container 6361ea1c4a8015455773d47f65b44428176ab5232640cc989fd96ec79863fad4: Status 404 returned error can't find the container with id 6361ea1c4a8015455773d47f65b44428176ab5232640cc989fd96ec79863fad4
Jul 16 22:23:06.291447 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.291408    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f30b043-2edb-400a-9a0f-e10350f67802" containerID="3ef5620a0b5d1431ef0f3b716af917573009265ff7672b875b162d325bd1b06f" exitCode=7
Jul 16 22:23:06.291607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.291489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" event={"ID":"8f30b043-2edb-400a-9a0f-e10350f67802","Type":"ContainerDied","Data":"3ef5620a0b5d1431ef0f3b716af917573009265ff7672b875b162d325bd1b06f"}
Jul 16 22:23:06.291607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.291539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" event={"ID":"8f30b043-2edb-400a-9a0f-e10350f67802","Type":"ContainerStarted","Data":"6361ea1c4a8015455773d47f65b44428176ab5232640cc989fd96ec79863fad4"}
Jul 16 22:23:06.291735 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.291719    2555 scope.go:117] "RemoveContainer" containerID="3ef5620a0b5d1431ef0f3b716af917573009265ff7672b875b162d325bd1b06f"
Jul 16 22:23:06.313300 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.313272    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"]
Jul 16 22:23:06.435745 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.435720    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"
Jul 16 22:23:06.545737 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.545695    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb4a71-36f5-49d6-8efe-378cdfb8a7e9-maas-api-service-ca\") pod \"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9\" (UID: \"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9\") "
Jul 16 22:23:06.546082 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.546056    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41eb4a71-36f5-49d6-8efe-378cdfb8a7e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" (UID: "41eb4a71-36f5-49d6-8efe-378cdfb8a7e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:06.646927 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:06.646886    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb4a71-36f5-49d6-8efe-378cdfb8a7e9-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:23:07.298918 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.298839    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f30b043-2edb-400a-9a0f-e10350f67802" containerID="6866bcb1157ec8dfea8ab83704f30e7747aa445cef1f96bf0b7d6b475ab1c8d2" exitCode=7
Jul 16 22:23:07.299101 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.298928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" event={"ID":"8f30b043-2edb-400a-9a0f-e10350f67802","Type":"ContainerDied","Data":"6866bcb1157ec8dfea8ab83704f30e7747aa445cef1f96bf0b7d6b475ab1c8d2"}
Jul 16 22:23:07.299101 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.299020    2555 scope.go:117] "RemoveContainer" containerID="3ef5620a0b5d1431ef0f3b716af917573009265ff7672b875b162d325bd1b06f"
Jul 16 22:23:07.299214 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.299139    2555 scope.go:117] "RemoveContainer" containerID="6866bcb1157ec8dfea8ab83704f30e7747aa445cef1f96bf0b7d6b475ab1c8d2"
Jul 16 22:23:07.299404 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:07.299374    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7_odh-ai-gateway-infra(8f30b043-2edb-400a-9a0f-e10350f67802)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" podUID="8f30b043-2edb-400a-9a0f-e10350f67802"
Jul 16 22:23:07.300264 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.300248    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"
Jul 16 22:23:07.300342 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.300247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r" event={"ID":"41eb4a71-36f5-49d6-8efe-378cdfb8a7e9","Type":"ContainerDied","Data":"d54de945c5015bef0a718cfaa85f12c54cd24a20d310f506b6d0290b3731448b"}
Jul 16 22:23:07.310304 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.310286    2555 scope.go:117] "RemoveContainer" containerID="274d852b876ab5d106291b3223a9480d175af8b860b57bd0e547e5c3b93cd1ce"
Jul 16 22:23:07.335303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.335271    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"]
Jul 16 22:23:07.341440 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:07.341415    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-z8j4r"]
Jul 16 22:23:08.308469 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:08.308438    2555 scope.go:117] "RemoveContainer" containerID="6866bcb1157ec8dfea8ab83704f30e7747aa445cef1f96bf0b7d6b475ab1c8d2"
Jul 16 22:23:08.308844 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:08.308667    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7_odh-ai-gateway-infra(8f30b043-2edb-400a-9a0f-e10350f67802)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" podUID="8f30b043-2edb-400a-9a0f-e10350f67802"
Jul 16 22:23:08.495046 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:08.495011    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" path="/var/lib/kubelet/pods/41eb4a71-36f5-49d6-8efe-378cdfb8a7e9/volumes"
Jul 16 22:23:23.490471 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:23.490395    2555 scope.go:117] "RemoveContainer" containerID="6866bcb1157ec8dfea8ab83704f30e7747aa445cef1f96bf0b7d6b475ab1c8d2"
Jul 16 22:23:24.374427 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:24.374396    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f30b043-2edb-400a-9a0f-e10350f67802" containerID="3fa7d2ea396073a9a25a31fbfa061f410a6287ba407570bcc4f1bd8852dd6a5f" exitCode=7
Jul 16 22:23:24.374591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:24.374470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" event={"ID":"8f30b043-2edb-400a-9a0f-e10350f67802","Type":"ContainerDied","Data":"3fa7d2ea396073a9a25a31fbfa061f410a6287ba407570bcc4f1bd8852dd6a5f"}
Jul 16 22:23:24.374591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:24.374527    2555 scope.go:117] "RemoveContainer" containerID="6866bcb1157ec8dfea8ab83704f30e7747aa445cef1f96bf0b7d6b475ab1c8d2"
Jul 16 22:23:24.374879 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:24.374865    2555 scope.go:117] "RemoveContainer" containerID="3fa7d2ea396073a9a25a31fbfa061f410a6287ba407570bcc4f1bd8852dd6a5f"
Jul 16 22:23:24.375081 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:24.375065    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7_odh-ai-gateway-infra(8f30b043-2edb-400a-9a0f-e10350f67802)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" podUID="8f30b043-2edb-400a-9a0f-e10350f67802"
Jul 16 22:23:25.398923 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:25.398889    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"]
Jul 16 22:23:25.536395 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:25.536369    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"
Jul 16 22:23:25.594751 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:25.594711    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f30b043-2edb-400a-9a0f-e10350f67802-maas-api-service-ca\") pod \"8f30b043-2edb-400a-9a0f-e10350f67802\" (UID: \"8f30b043-2edb-400a-9a0f-e10350f67802\") "
Jul 16 22:23:25.595116 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:25.595092    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f30b043-2edb-400a-9a0f-e10350f67802-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f30b043-2edb-400a-9a0f-e10350f67802" (UID: "8f30b043-2edb-400a-9a0f-e10350f67802"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:25.696315 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:25.696231    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f30b043-2edb-400a-9a0f-e10350f67802-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:23:26.385559 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:26.385527    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7" event={"ID":"8f30b043-2edb-400a-9a0f-e10350f67802","Type":"ContainerDied","Data":"6361ea1c4a8015455773d47f65b44428176ab5232640cc989fd96ec79863fad4"}
Jul 16 22:23:26.385559 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:26.385557    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"
Jul 16 22:23:26.385812 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:26.385587    2555 scope.go:117] "RemoveContainer" containerID="3fa7d2ea396073a9a25a31fbfa061f410a6287ba407570bcc4f1bd8852dd6a5f"
Jul 16 22:23:26.406562 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:26.406539    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"]
Jul 16 22:23:26.411303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:26.411281    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-svhm7"]
Jul 16 22:23:26.495024 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:26.494991    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" path="/var/lib/kubelet/pods/8f30b043-2edb-400a-9a0f-e10350f67802/volumes"
Jul 16 22:23:34.770651 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770616    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"]
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770902    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770912    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770923    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770928    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770944    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770949    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770978    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770987    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.770997    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.771002    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e0a7cc-5417-4436-9ccb-2729599440bf" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.771010    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.771015    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.771057    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.771063    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.771071    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:34.771146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.771078    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:34.773944 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.773927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"
Jul 16 22:23:34.776754 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.776735    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:23:34.776862 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.776781    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:23:34.781811 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.781788    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"]
Jul 16 22:23:34.881216 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.881170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c161d4b1-d771-449f-9594-0ba6f3ec7cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-qtflb\" (UID: \"c161d4b1-d771-449f-9594-0ba6f3ec7cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"
Jul 16 22:23:34.982594 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.982548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c161d4b1-d771-449f-9594-0ba6f3ec7cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-qtflb\" (UID: \"c161d4b1-d771-449f-9594-0ba6f3ec7cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"
Jul 16 22:23:34.983263 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:34.983245    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c161d4b1-d771-449f-9594-0ba6f3ec7cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-qtflb\" (UID: \"c161d4b1-d771-449f-9594-0ba6f3ec7cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"
Jul 16 22:23:35.084448 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:35.084411    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"
Jul 16 22:23:35.217090 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:35.217039    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"]
Jul 16 22:23:35.219258 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:23:35.219229    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc161d4b1_d771_449f_9594_0ba6f3ec7cf2.slice/crio-514c46de2d8a87f0b645ccbaa6b8ddd706b0c4c3f592504d85528f01c2a9ff92 WatchSource:0}: Error finding container 514c46de2d8a87f0b645ccbaa6b8ddd706b0c4c3f592504d85528f01c2a9ff92: Status 404 returned error can't find the container with id 514c46de2d8a87f0b645ccbaa6b8ddd706b0c4c3f592504d85528f01c2a9ff92
Jul 16 22:23:35.423190 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:35.423102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" event={"ID":"c161d4b1-d771-449f-9594-0ba6f3ec7cf2","Type":"ContainerStarted","Data":"514c46de2d8a87f0b645ccbaa6b8ddd706b0c4c3f592504d85528f01c2a9ff92"}
Jul 16 22:23:36.428643 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:36.428608    2555 generic.go:356] "Generic (PLEG): container finished" podID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerID="d21455afed1254f807cc2a247a66b034c8cf7c1b24ab03a801150c9c99720241" exitCode=6
Jul 16 22:23:36.429109 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:36.428701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" event={"ID":"c161d4b1-d771-449f-9594-0ba6f3ec7cf2","Type":"ContainerDied","Data":"d21455afed1254f807cc2a247a66b034c8cf7c1b24ab03a801150c9c99720241"}
Jul 16 22:23:36.429109 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:36.428857    2555 scope.go:117] "RemoveContainer" containerID="d21455afed1254f807cc2a247a66b034c8cf7c1b24ab03a801150c9c99720241"
Jul 16 22:23:37.434269 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:37.434233    2555 generic.go:356] "Generic (PLEG): container finished" podID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerID="231ed52eb5b1270cde03a852e681a19bfd47dcd22218d8ec6fa50246d5273bb8" exitCode=6
Jul 16 22:23:37.434680 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:37.434321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" event={"ID":"c161d4b1-d771-449f-9594-0ba6f3ec7cf2","Type":"ContainerDied","Data":"231ed52eb5b1270cde03a852e681a19bfd47dcd22218d8ec6fa50246d5273bb8"}
Jul 16 22:23:37.434680 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:37.434387    2555 scope.go:117] "RemoveContainer" containerID="d21455afed1254f807cc2a247a66b034c8cf7c1b24ab03a801150c9c99720241"
Jul 16 22:23:37.434680 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:37.434557    2555 scope.go:117] "RemoveContainer" containerID="231ed52eb5b1270cde03a852e681a19bfd47dcd22218d8ec6fa50246d5273bb8"
Jul 16 22:23:37.434834 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:37.434777    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-qtflb_odh-ai-gateway-infra(c161d4b1-d771-449f-9594-0ba6f3ec7cf2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2"
Jul 16 22:23:38.439895 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:38.439871    2555 scope.go:117] "RemoveContainer" containerID="231ed52eb5b1270cde03a852e681a19bfd47dcd22218d8ec6fa50246d5273bb8"
Jul 16 22:23:38.440287 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:38.440086    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-qtflb_odh-ai-gateway-infra(c161d4b1-d771-449f-9594-0ba6f3ec7cf2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2"
Jul 16 22:23:46.502898 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.502866    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"]
Jul 16 22:23:46.503264 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.503166    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:46.503264 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.503177    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:46.503264 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.503233    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f30b043-2edb-400a-9a0f-e10350f67802" containerName="revoke-keys"
Jul 16 22:23:46.503264 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.503244    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="41eb4a71-36f5-49d6-8efe-378cdfb8a7e9" containerName="revoke-keys"
Jul 16 22:23:46.507360 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.507343    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"
Jul 16 22:23:46.512847 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.512819    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"]
Jul 16 22:23:46.579796 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.579764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e896aa-97b2-4949-bfdc-1c2f05860d2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm\" (UID: \"e8e896aa-97b2-4949-bfdc-1c2f05860d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"
Jul 16 22:23:46.680321 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.680291    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e896aa-97b2-4949-bfdc-1c2f05860d2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm\" (UID: \"e8e896aa-97b2-4949-bfdc-1c2f05860d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"
Jul 16 22:23:46.680938 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.680917    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e896aa-97b2-4949-bfdc-1c2f05860d2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm\" (UID: \"e8e896aa-97b2-4949-bfdc-1c2f05860d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"
Jul 16 22:23:46.819917 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:46.819812    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"
Jul 16 22:23:47.166439 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:47.166295    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"]
Jul 16 22:23:47.169302 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:23:47.169264    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e896aa_97b2_4949_bfdc_1c2f05860d2f.slice/crio-9b49cdeeeed43a79a79cfb7d465d0f30db20fa165f7615d2bfda6536b0e173e8 WatchSource:0}: Error finding container 9b49cdeeeed43a79a79cfb7d465d0f30db20fa165f7615d2bfda6536b0e173e8: Status 404 returned error can't find the container with id 9b49cdeeeed43a79a79cfb7d465d0f30db20fa165f7615d2bfda6536b0e173e8
Jul 16 22:23:47.478034 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:47.477991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" event={"ID":"e8e896aa-97b2-4949-bfdc-1c2f05860d2f","Type":"ContainerStarted","Data":"7511f1db4d6d49b4557c7308095c736240f0cbff22424da34b9d387cf702e7b3"}
Jul 16 22:23:47.478190 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:47.478093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" event={"ID":"e8e896aa-97b2-4949-bfdc-1c2f05860d2f","Type":"ContainerStarted","Data":"9b49cdeeeed43a79a79cfb7d465d0f30db20fa165f7615d2bfda6536b0e173e8"}
Jul 16 22:23:47.498635 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:47.498546    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" podStartSLOduration=1.498526803 podStartE2EDuration="1.498526803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:47.497368007 +0000 UTC m=+2183.490532170" watchObservedRunningTime="2026-07-16 22:23:47.498526803 +0000 UTC m=+2183.491690955"
Jul 16 22:23:48.483259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:48.483227    2555 generic.go:356] "Generic (PLEG): container finished" podID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerID="7511f1db4d6d49b4557c7308095c736240f0cbff22424da34b9d387cf702e7b3" exitCode=6
Jul 16 22:23:48.483660 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:48.483314    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" event={"ID":"e8e896aa-97b2-4949-bfdc-1c2f05860d2f","Type":"ContainerDied","Data":"7511f1db4d6d49b4557c7308095c736240f0cbff22424da34b9d387cf702e7b3"}
Jul 16 22:23:48.483660 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:48.483471    2555 scope.go:117] "RemoveContainer" containerID="7511f1db4d6d49b4557c7308095c736240f0cbff22424da34b9d387cf702e7b3"
Jul 16 22:23:49.489113 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:49.489078    2555 generic.go:356] "Generic (PLEG): container finished" podID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerID="0067a2e3e5df96c44676cf160a115cf7fd83b65b0bfa97aeae4c0f8fc15057f7" exitCode=6
Jul 16 22:23:49.489591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:49.489166    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" event={"ID":"e8e896aa-97b2-4949-bfdc-1c2f05860d2f","Type":"ContainerDied","Data":"0067a2e3e5df96c44676cf160a115cf7fd83b65b0bfa97aeae4c0f8fc15057f7"}
Jul 16 22:23:49.489591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:49.489216    2555 scope.go:117] "RemoveContainer" containerID="7511f1db4d6d49b4557c7308095c736240f0cbff22424da34b9d387cf702e7b3"
Jul 16 22:23:49.489591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:49.489390    2555 scope.go:117] "RemoveContainer" containerID="0067a2e3e5df96c44676cf160a115cf7fd83b65b0bfa97aeae4c0f8fc15057f7"
Jul 16 22:23:49.489591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:49.489463    2555 scope.go:117] "RemoveContainer" containerID="231ed52eb5b1270cde03a852e681a19bfd47dcd22218d8ec6fa50246d5273bb8"
Jul 16 22:23:49.489806 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:49.489651    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm_odh-ai-gateway-infra(e8e896aa-97b2-4949-bfdc-1c2f05860d2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f"
Jul 16 22:23:50.499933 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:50.499902    2555 generic.go:356] "Generic (PLEG): container finished" podID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerID="4b4353de644584f32397c9b9e037d598981bf5dc11d0a8709644f19abb3f623e" exitCode=6
Jul 16 22:23:50.500419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:50.499985    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" event={"ID":"c161d4b1-d771-449f-9594-0ba6f3ec7cf2","Type":"ContainerDied","Data":"4b4353de644584f32397c9b9e037d598981bf5dc11d0a8709644f19abb3f623e"}
Jul 16 22:23:50.500419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:50.500044    2555 scope.go:117] "RemoveContainer" containerID="231ed52eb5b1270cde03a852e681a19bfd47dcd22218d8ec6fa50246d5273bb8"
Jul 16 22:23:50.500419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:50.500353    2555 scope.go:117] "RemoveContainer" containerID="4b4353de644584f32397c9b9e037d598981bf5dc11d0a8709644f19abb3f623e"
Jul 16 22:23:50.500602 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:50.500554    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-qtflb_odh-ai-gateway-infra(c161d4b1-d771-449f-9594-0ba6f3ec7cf2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2"
Jul 16 22:23:50.501811 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:50.501792    2555 scope.go:117] "RemoveContainer" containerID="0067a2e3e5df96c44676cf160a115cf7fd83b65b0bfa97aeae4c0f8fc15057f7"
Jul 16 22:23:50.502055 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:23:50.502033    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm_odh-ai-gateway-infra(e8e896aa-97b2-4949-bfdc-1c2f05860d2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f"
Jul 16 22:23:51.526658 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:51.526627    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"]
Jul 16 22:23:51.654712 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:51.654688    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"
Jul 16 22:23:51.725341 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:51.725309    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c161d4b1-d771-449f-9594-0ba6f3ec7cf2-maas-api-service-ca\") pod \"c161d4b1-d771-449f-9594-0ba6f3ec7cf2\" (UID: \"c161d4b1-d771-449f-9594-0ba6f3ec7cf2\") "
Jul 16 22:23:51.725652 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:51.725630    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c161d4b1-d771-449f-9594-0ba6f3ec7cf2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c161d4b1-d771-449f-9594-0ba6f3ec7cf2" (UID: "c161d4b1-d771-449f-9594-0ba6f3ec7cf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:51.826036 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:51.826000    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c161d4b1-d771-449f-9594-0ba6f3ec7cf2-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:23:52.512714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:52.512688    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"
Jul 16 22:23:52.512714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:52.512705    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb" event={"ID":"c161d4b1-d771-449f-9594-0ba6f3ec7cf2","Type":"ContainerDied","Data":"514c46de2d8a87f0b645ccbaa6b8ddd706b0c4c3f592504d85528f01c2a9ff92"}
Jul 16 22:23:52.512919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:52.512759    2555 scope.go:117] "RemoveContainer" containerID="4b4353de644584f32397c9b9e037d598981bf5dc11d0a8709644f19abb3f623e"
Jul 16 22:23:52.529272 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:52.529247    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"]
Jul 16 22:23:52.533665 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:52.533641    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-qtflb"]
Jul 16 22:23:54.494607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:23:54.494576    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" path="/var/lib/kubelet/pods/c161d4b1-d771-449f-9594-0ba6f3ec7cf2/volumes"
Jul 16 22:24:05.139051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139013    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"]
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139319    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139330    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139339    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139344    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139352    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139357    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139401    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139409    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.139437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.139415    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c161d4b1-d771-449f-9594-0ba6f3ec7cf2" containerName="revoke-keys"
Jul 16 22:24:05.141749 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.141733    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"
Jul 16 22:24:05.149805 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.149779    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"]
Jul 16 22:24:05.247744 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.247713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283e00b-790e-43e0-9ddd-bf08432e7bee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm\" (UID: \"0283e00b-790e-43e0-9ddd-bf08432e7bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"
Jul 16 22:24:05.348871 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.348827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283e00b-790e-43e0-9ddd-bf08432e7bee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm\" (UID: \"0283e00b-790e-43e0-9ddd-bf08432e7bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"
Jul 16 22:24:05.349505 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.349480    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283e00b-790e-43e0-9ddd-bf08432e7bee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm\" (UID: \"0283e00b-790e-43e0-9ddd-bf08432e7bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"
Jul 16 22:24:05.453081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.452983    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"
Jul 16 22:24:05.490028 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.489998    2555 scope.go:117] "RemoveContainer" containerID="0067a2e3e5df96c44676cf160a115cf7fd83b65b0bfa97aeae4c0f8fc15057f7"
Jul 16 22:24:05.589456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:05.589241    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"]
Jul 16 22:24:05.591690 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:24:05.591656    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0283e00b_790e_43e0_9ddd_bf08432e7bee.slice/crio-f5b5b64bec176c8bbb9e13ad4b29e972961f7dbad852b9d2a83f4cb9008c062e WatchSource:0}: Error finding container f5b5b64bec176c8bbb9e13ad4b29e972961f7dbad852b9d2a83f4cb9008c062e: Status 404 returned error can't find the container with id f5b5b64bec176c8bbb9e13ad4b29e972961f7dbad852b9d2a83f4cb9008c062e
Jul 16 22:24:06.574075 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.574045    2555 generic.go:356] "Generic (PLEG): container finished" podID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerID="ff1caa1585b8dfc438be8a25c8d458083b0079c0ede849241d4e2cf404a4165c" exitCode=6
Jul 16 22:24:06.574719 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.574119    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" event={"ID":"e8e896aa-97b2-4949-bfdc-1c2f05860d2f","Type":"ContainerDied","Data":"ff1caa1585b8dfc438be8a25c8d458083b0079c0ede849241d4e2cf404a4165c"}
Jul 16 22:24:06.574719 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.574179    2555 scope.go:117] "RemoveContainer" containerID="0067a2e3e5df96c44676cf160a115cf7fd83b65b0bfa97aeae4c0f8fc15057f7"
Jul 16 22:24:06.574719 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.574451    2555 scope.go:117] "RemoveContainer" containerID="ff1caa1585b8dfc438be8a25c8d458083b0079c0ede849241d4e2cf404a4165c"
Jul 16 22:24:06.574719 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:06.574677    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm_odh-ai-gateway-infra(e8e896aa-97b2-4949-bfdc-1c2f05860d2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f"
Jul 16 22:24:06.575767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.575744    2555 generic.go:356] "Generic (PLEG): container finished" podID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerID="ab1ee4b642a028cbdc475deb8235d111576a54eb56cf0715e97709be41548195" exitCode=7
Jul 16 22:24:06.575849 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.575779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" event={"ID":"0283e00b-790e-43e0-9ddd-bf08432e7bee","Type":"ContainerDied","Data":"ab1ee4b642a028cbdc475deb8235d111576a54eb56cf0715e97709be41548195"}
Jul 16 22:24:06.575849 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.575817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" event={"ID":"0283e00b-790e-43e0-9ddd-bf08432e7bee","Type":"ContainerStarted","Data":"f5b5b64bec176c8bbb9e13ad4b29e972961f7dbad852b9d2a83f4cb9008c062e"}
Jul 16 22:24:06.576030 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:06.576016    2555 scope.go:117] "RemoveContainer" containerID="ab1ee4b642a028cbdc475deb8235d111576a54eb56cf0715e97709be41548195"
Jul 16 22:24:07.583183 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.583143    2555 generic.go:356] "Generic (PLEG): container finished" podID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerID="a6ad109915b89fa5f74880ce0f4f067afc64d26ba82bfdd3005f896fb98157e9" exitCode=7
Jul 16 22:24:07.583634 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.583229    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" event={"ID":"0283e00b-790e-43e0-9ddd-bf08432e7bee","Type":"ContainerDied","Data":"a6ad109915b89fa5f74880ce0f4f067afc64d26ba82bfdd3005f896fb98157e9"}
Jul 16 22:24:07.583634 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.583283    2555 scope.go:117] "RemoveContainer" containerID="ab1ee4b642a028cbdc475deb8235d111576a54eb56cf0715e97709be41548195"
Jul 16 22:24:07.583634 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.583469    2555 scope.go:117] "RemoveContainer" containerID="a6ad109915b89fa5f74880ce0f4f067afc64d26ba82bfdd3005f896fb98157e9"
Jul 16 22:24:07.583781 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:07.583654    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm_odh-ai-gateway-infra(0283e00b-790e-43e0-9ddd-bf08432e7bee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee"
Jul 16 22:24:07.603120 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.603094    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"]
Jul 16 22:24:07.734155 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.734128    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"
Jul 16 22:24:07.765061 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.765034    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e896aa-97b2-4949-bfdc-1c2f05860d2f-maas-api-service-ca\") pod \"e8e896aa-97b2-4949-bfdc-1c2f05860d2f\" (UID: \"e8e896aa-97b2-4949-bfdc-1c2f05860d2f\") "
Jul 16 22:24:07.765391 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.765370    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e896aa-97b2-4949-bfdc-1c2f05860d2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8e896aa-97b2-4949-bfdc-1c2f05860d2f" (UID: "e8e896aa-97b2-4949-bfdc-1c2f05860d2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:07.871491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:07.866790    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e896aa-97b2-4949-bfdc-1c2f05860d2f-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:24:08.588740 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:08.588705    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"
Jul 16 22:24:08.589269 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:08.588707    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm" event={"ID":"e8e896aa-97b2-4949-bfdc-1c2f05860d2f","Type":"ContainerDied","Data":"9b49cdeeeed43a79a79cfb7d465d0f30db20fa165f7615d2bfda6536b0e173e8"}
Jul 16 22:24:08.589269 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:08.588851    2555 scope.go:117] "RemoveContainer" containerID="ff1caa1585b8dfc438be8a25c8d458083b0079c0ede849241d4e2cf404a4165c"
Jul 16 22:24:08.590682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:08.590657    2555 scope.go:117] "RemoveContainer" containerID="a6ad109915b89fa5f74880ce0f4f067afc64d26ba82bfdd3005f896fb98157e9"
Jul 16 22:24:08.590908 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:08.590883    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm_odh-ai-gateway-infra(0283e00b-790e-43e0-9ddd-bf08432e7bee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee"
Jul 16 22:24:08.625862 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:08.625829    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"]
Jul 16 22:24:08.629692 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:08.629668    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xnhsm"]
Jul 16 22:24:10.494422 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:10.494392    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" path="/var/lib/kubelet/pods/e8e896aa-97b2-4949-bfdc-1c2f05860d2f/volumes"
Jul 16 22:24:19.490197 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:19.490163    2555 scope.go:117] "RemoveContainer" containerID="a6ad109915b89fa5f74880ce0f4f067afc64d26ba82bfdd3005f896fb98157e9"
Jul 16 22:24:20.641637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:20.641601    2555 generic.go:356] "Generic (PLEG): container finished" podID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerID="1c28adaf20ca3fcca7037ac3cd883fa4cffa4329c6eaf4404dbfa59058c52b3d" exitCode=7
Jul 16 22:24:20.642152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:20.641643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" event={"ID":"0283e00b-790e-43e0-9ddd-bf08432e7bee","Type":"ContainerDied","Data":"1c28adaf20ca3fcca7037ac3cd883fa4cffa4329c6eaf4404dbfa59058c52b3d"}
Jul 16 22:24:20.642152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:20.641705    2555 scope.go:117] "RemoveContainer" containerID="a6ad109915b89fa5f74880ce0f4f067afc64d26ba82bfdd3005f896fb98157e9"
Jul 16 22:24:20.642152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:20.641988    2555 scope.go:117] "RemoveContainer" containerID="1c28adaf20ca3fcca7037ac3cd883fa4cffa4329c6eaf4404dbfa59058c52b3d"
Jul 16 22:24:20.642342 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:20.642197    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm_odh-ai-gateway-infra(0283e00b-790e-43e0-9ddd-bf08432e7bee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee"
Jul 16 22:24:21.708306 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:21.708272    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"]
Jul 16 22:24:21.838872 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:21.838847    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"
Jul 16 22:24:21.883201 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:21.883167    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283e00b-790e-43e0-9ddd-bf08432e7bee-maas-api-service-ca\") pod \"0283e00b-790e-43e0-9ddd-bf08432e7bee\" (UID: \"0283e00b-790e-43e0-9ddd-bf08432e7bee\") "
Jul 16 22:24:21.883568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:21.883547    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283e00b-790e-43e0-9ddd-bf08432e7bee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0283e00b-790e-43e0-9ddd-bf08432e7bee" (UID: "0283e00b-790e-43e0-9ddd-bf08432e7bee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:21.984459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:21.984375    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283e00b-790e-43e0-9ddd-bf08432e7bee-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:24:22.655609 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:22.655571    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm" event={"ID":"0283e00b-790e-43e0-9ddd-bf08432e7bee","Type":"ContainerDied","Data":"f5b5b64bec176c8bbb9e13ad4b29e972961f7dbad852b9d2a83f4cb9008c062e"}
Jul 16 22:24:22.655609 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:22.655600    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"
Jul 16 22:24:22.655826 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:22.655644    2555 scope.go:117] "RemoveContainer" containerID="1c28adaf20ca3fcca7037ac3cd883fa4cffa4329c6eaf4404dbfa59058c52b3d"
Jul 16 22:24:22.672590 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:22.672564    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"]
Jul 16 22:24:22.679002 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:22.678978    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-g6pdm"]
Jul 16 22:24:24.495713 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.495680    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" path="/var/lib/kubelet/pods/0283e00b-790e-43e0-9ddd-bf08432e7bee/volumes"
Jul 16 22:24:24.818077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818002    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"]
Jul 16 22:24:24.818315 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818302    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:24.818369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818316    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:24.818369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:24.818369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818331    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:24.818369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818338    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:24.818369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818343    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:24.818369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818352    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:24.818369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818357    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:24.818602 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818414    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:24.818602 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818421    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:24.818602 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:24.818602 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818434    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:24.818602 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.818439    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:24.822649 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.822630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"
Jul 16 22:24:24.825496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.825476    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:24:24.825614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.825520    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:24:24.829919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.829227    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"]
Jul 16 22:24:24.909785 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:24.909751    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cb2844-5d50-4c0a-a06c-9bc0a2272d92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b\" (UID: \"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"
Jul 16 22:24:25.010869 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:25.010828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cb2844-5d50-4c0a-a06c-9bc0a2272d92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b\" (UID: \"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"
Jul 16 22:24:25.011453 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:25.011431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cb2844-5d50-4c0a-a06c-9bc0a2272d92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b\" (UID: \"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"
Jul 16 22:24:25.133600 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:25.133560    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"
Jul 16 22:24:25.275507 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:25.275481    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"]
Jul 16 22:24:25.277903 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:24:25.277868    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cb2844_5d50_4c0a_a06c_9bc0a2272d92.slice/crio-e59a216e6b67efc5faf0c9dbd47822cdf3d1c48b55a7e115023c041882a175f9 WatchSource:0}: Error finding container e59a216e6b67efc5faf0c9dbd47822cdf3d1c48b55a7e115023c041882a175f9: Status 404 returned error can't find the container with id e59a216e6b67efc5faf0c9dbd47822cdf3d1c48b55a7e115023c041882a175f9
Jul 16 22:24:25.670874 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:25.670840    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" event={"ID":"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92","Type":"ContainerStarted","Data":"e559386c381a89272f65ee32cdea6c08735534b4f0582e2b3f662987cfbd1fad"}
Jul 16 22:24:25.671246 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:25.670896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" event={"ID":"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92","Type":"ContainerStarted","Data":"e59a216e6b67efc5faf0c9dbd47822cdf3d1c48b55a7e115023c041882a175f9"}
Jul 16 22:24:25.687027 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:25.686981    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" podStartSLOduration=1.686948745 podStartE2EDuration="1.686948745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:25.685750972 +0000 UTC m=+2221.678915124" watchObservedRunningTime="2026-07-16 22:24:25.686948745 +0000 UTC m=+2221.680112896"
Jul 16 22:24:26.676842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:26.676805    2555 generic.go:356] "Generic (PLEG): container finished" podID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerID="e559386c381a89272f65ee32cdea6c08735534b4f0582e2b3f662987cfbd1fad" exitCode=6
Jul 16 22:24:26.677296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:26.676870    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" event={"ID":"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92","Type":"ContainerDied","Data":"e559386c381a89272f65ee32cdea6c08735534b4f0582e2b3f662987cfbd1fad"}
Jul 16 22:24:26.677296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:26.677133    2555 scope.go:117] "RemoveContainer" containerID="e559386c381a89272f65ee32cdea6c08735534b4f0582e2b3f662987cfbd1fad"
Jul 16 22:24:27.682441 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:27.682408    2555 generic.go:356] "Generic (PLEG): container finished" podID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerID="aeeacb38daf982286140a17a7c8c9fc760e5ad533dad2536ee1e5d8a21ef9828" exitCode=6
Jul 16 22:24:27.682441 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:27.682446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" event={"ID":"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92","Type":"ContainerDied","Data":"aeeacb38daf982286140a17a7c8c9fc760e5ad533dad2536ee1e5d8a21ef9828"}
Jul 16 22:24:27.682929 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:27.682497    2555 scope.go:117] "RemoveContainer" containerID="e559386c381a89272f65ee32cdea6c08735534b4f0582e2b3f662987cfbd1fad"
Jul 16 22:24:27.682929 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:27.682685    2555 scope.go:117] "RemoveContainer" containerID="aeeacb38daf982286140a17a7c8c9fc760e5ad533dad2536ee1e5d8a21ef9828"
Jul 16 22:24:27.682929 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:27.682912    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b_odh-ai-gateway-infra(f7cb2844-5d50-4c0a-a06c-9bc0a2272d92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92"
Jul 16 22:24:28.687882 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:28.687858    2555 scope.go:117] "RemoveContainer" containerID="aeeacb38daf982286140a17a7c8c9fc760e5ad533dad2536ee1e5d8a21ef9828"
Jul 16 22:24:28.688267 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:28.688075    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b_odh-ai-gateway-infra(f7cb2844-5d50-4c0a-a06c-9bc0a2272d92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92"
Jul 16 22:24:42.489931 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:42.489898    2555 scope.go:117] "RemoveContainer" containerID="aeeacb38daf982286140a17a7c8c9fc760e5ad533dad2536ee1e5d8a21ef9828"
Jul 16 22:24:42.745679 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:42.745583    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" event={"ID":"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92","Type":"ContainerStarted","Data":"44073a9a172bfb475b142e41c4d29474a6d5f2557006ef879022da6b8b655ad7"}
Jul 16 22:24:43.751549 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:43.751518    2555 generic.go:356] "Generic (PLEG): container finished" podID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerID="44073a9a172bfb475b142e41c4d29474a6d5f2557006ef879022da6b8b655ad7" exitCode=6
Jul 16 22:24:43.752057 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:43.751585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" event={"ID":"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92","Type":"ContainerDied","Data":"44073a9a172bfb475b142e41c4d29474a6d5f2557006ef879022da6b8b655ad7"}
Jul 16 22:24:43.752057 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:43.751659    2555 scope.go:117] "RemoveContainer" containerID="aeeacb38daf982286140a17a7c8c9fc760e5ad533dad2536ee1e5d8a21ef9828"
Jul 16 22:24:43.752057 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:43.751933    2555 scope.go:117] "RemoveContainer" containerID="44073a9a172bfb475b142e41c4d29474a6d5f2557006ef879022da6b8b655ad7"
Jul 16 22:24:43.752248 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:43.752226    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b_odh-ai-gateway-infra(f7cb2844-5d50-4c0a-a06c-9bc0a2272d92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92"
Jul 16 22:24:43.777548 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:43.777522    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"]
Jul 16 22:24:44.884628 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:44.884603    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"
Jul 16 22:24:44.985616 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:44.985577    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cb2844-5d50-4c0a-a06c-9bc0a2272d92-maas-api-service-ca\") pod \"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92\" (UID: \"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92\") "
Jul 16 22:24:44.985948 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:44.985923    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cb2844-5d50-4c0a-a06c-9bc0a2272d92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" (UID: "f7cb2844-5d50-4c0a-a06c-9bc0a2272d92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:45.086719 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:45.086683    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cb2844-5d50-4c0a-a06c-9bc0a2272d92-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:24:45.762575 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:45.762545    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"
Jul 16 22:24:45.762575 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:45.762567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b" event={"ID":"f7cb2844-5d50-4c0a-a06c-9bc0a2272d92","Type":"ContainerDied","Data":"e59a216e6b67efc5faf0c9dbd47822cdf3d1c48b55a7e115023c041882a175f9"}
Jul 16 22:24:45.762807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:45.762621    2555 scope.go:117] "RemoveContainer" containerID="44073a9a172bfb475b142e41c4d29474a6d5f2557006ef879022da6b8b655ad7"
Jul 16 22:24:45.782946 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:45.782921    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"]
Jul 16 22:24:45.788947 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:45.788921    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-2ll6b"]
Jul 16 22:24:46.494926 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.494892    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" path="/var/lib/kubelet/pods/f7cb2844-5d50-4c0a-a06c-9bc0a2272d92/volumes"
Jul 16 22:24:46.552891 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.552861    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"]
Jul 16 22:24:46.553221 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553207    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:46.553278 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553223    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:46.553278 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553232    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:46.553278 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553237    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:46.553278 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553244    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:46.553278 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553249    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e896aa-97b2-4949-bfdc-1c2f05860d2f" containerName="revoke-keys"
Jul 16 22:24:46.553278 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553257    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:46.553278 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553263    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:46.553515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553321    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0283e00b-790e-43e0-9ddd-bf08432e7bee" containerName="revoke-keys"
Jul 16 22:24:46.553515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553328    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:46.553515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.553333    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:46.557696 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.557677    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"
Jul 16 22:24:46.560458 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.560432    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:24:46.560691 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.560578    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:24:46.563939 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.563921    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"]
Jul 16 22:24:46.701118 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.701084    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70fb956-374c-48b2-b09b-6ad9f0d373b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs\" (UID: \"c70fb956-374c-48b2-b09b-6ad9f0d373b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"
Jul 16 22:24:46.801774 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.801670    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70fb956-374c-48b2-b09b-6ad9f0d373b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs\" (UID: \"c70fb956-374c-48b2-b09b-6ad9f0d373b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"
Jul 16 22:24:46.802370 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.802352    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70fb956-374c-48b2-b09b-6ad9f0d373b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs\" (UID: \"c70fb956-374c-48b2-b09b-6ad9f0d373b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"
Jul 16 22:24:46.869004 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.868946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"
Jul 16 22:24:46.995576 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:46.995550    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"]
Jul 16 22:24:46.997277 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:24:46.997246    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70fb956_374c_48b2_b09b_6ad9f0d373b5.slice/crio-e9dd57f8b2971281ca23a0a9b93813d8c49eec44a980c0dd7846fa5df370fb6d WatchSource:0}: Error finding container e9dd57f8b2971281ca23a0a9b93813d8c49eec44a980c0dd7846fa5df370fb6d: Status 404 returned error can't find the container with id e9dd57f8b2971281ca23a0a9b93813d8c49eec44a980c0dd7846fa5df370fb6d
Jul 16 22:24:47.773552 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:47.773510    2555 generic.go:356] "Generic (PLEG): container finished" podID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerID="5a6d53c54420f280478e70fc3d9d9ddced990b9fb37853c278a17267537af505" exitCode=6
Jul 16 22:24:47.773932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:47.773581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" event={"ID":"c70fb956-374c-48b2-b09b-6ad9f0d373b5","Type":"ContainerDied","Data":"5a6d53c54420f280478e70fc3d9d9ddced990b9fb37853c278a17267537af505"}
Jul 16 22:24:47.773932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:47.773637    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" event={"ID":"c70fb956-374c-48b2-b09b-6ad9f0d373b5","Type":"ContainerStarted","Data":"e9dd57f8b2971281ca23a0a9b93813d8c49eec44a980c0dd7846fa5df370fb6d"}
Jul 16 22:24:47.773932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:47.773766    2555 scope.go:117] "RemoveContainer" containerID="5a6d53c54420f280478e70fc3d9d9ddced990b9fb37853c278a17267537af505"
Jul 16 22:24:48.779054 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:48.778941    2555 generic.go:356] "Generic (PLEG): container finished" podID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerID="454a907b66c6ddd452ff6e901ba16e934121a3d598eff5e6797b067d8d77060b" exitCode=6
Jul 16 22:24:48.779054 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:48.779028    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" event={"ID":"c70fb956-374c-48b2-b09b-6ad9f0d373b5","Type":"ContainerDied","Data":"454a907b66c6ddd452ff6e901ba16e934121a3d598eff5e6797b067d8d77060b"}
Jul 16 22:24:48.779569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:48.779083    2555 scope.go:117] "RemoveContainer" containerID="5a6d53c54420f280478e70fc3d9d9ddced990b9fb37853c278a17267537af505"
Jul 16 22:24:48.779569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:48.779282    2555 scope.go:117] "RemoveContainer" containerID="454a907b66c6ddd452ff6e901ba16e934121a3d598eff5e6797b067d8d77060b"
Jul 16 22:24:48.779569 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:48.779529    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs_odh-ai-gateway-infra(c70fb956-374c-48b2-b09b-6ad9f0d373b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5"
Jul 16 22:24:49.785100 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:49.785067    2555 scope.go:117] "RemoveContainer" containerID="454a907b66c6ddd452ff6e901ba16e934121a3d598eff5e6797b067d8d77060b"
Jul 16 22:24:49.785496 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:49.785242    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs_odh-ai-gateway-infra(c70fb956-374c-48b2-b09b-6ad9f0d373b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5"
Jul 16 22:24:55.181937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.181898    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"]
Jul 16 22:24:55.182327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.182211    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:55.182327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.182222    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:55.182327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.182280    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cb2844-5d50-4c0a-a06c-9bc0a2272d92" containerName="revoke-keys"
Jul 16 22:24:55.186297 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.186279    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"
Jul 16 22:24:55.192800 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.192764    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"]
Jul 16 22:24:55.277077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.277032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d79c9-9deb-4843-b65f-3b0ac8681d62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25\" (UID: \"f11d79c9-9deb-4843-b65f-3b0ac8681d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"
Jul 16 22:24:55.377774 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.377740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d79c9-9deb-4843-b65f-3b0ac8681d62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25\" (UID: \"f11d79c9-9deb-4843-b65f-3b0ac8681d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"
Jul 16 22:24:55.378450 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.378430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d79c9-9deb-4843-b65f-3b0ac8681d62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25\" (UID: \"f11d79c9-9deb-4843-b65f-3b0ac8681d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"
Jul 16 22:24:55.497929 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.497833    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"
Jul 16 22:24:55.624043 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.623914    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"]
Jul 16 22:24:55.626724 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:24:55.626695    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11d79c9_9deb_4843_b65f_3b0ac8681d62.slice/crio-234f8cc4e58e9a5760a977eec78fea8f52750a705426a7b8b8d0f516f467435f WatchSource:0}: Error finding container 234f8cc4e58e9a5760a977eec78fea8f52750a705426a7b8b8d0f516f467435f: Status 404 returned error can't find the container with id 234f8cc4e58e9a5760a977eec78fea8f52750a705426a7b8b8d0f516f467435f
Jul 16 22:24:55.810384 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:55.810295    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" event={"ID":"f11d79c9-9deb-4843-b65f-3b0ac8681d62","Type":"ContainerStarted","Data":"234f8cc4e58e9a5760a977eec78fea8f52750a705426a7b8b8d0f516f467435f"}
Jul 16 22:24:56.815921 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:56.815886    2555 generic.go:356] "Generic (PLEG): container finished" podID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerID="e85ba2574cbe1d4bfd61ab24aea6322fdb096cef3ad7cfeff2bcb09d87939b06" exitCode=7
Jul 16 22:24:56.816308 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:56.815935    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" event={"ID":"f11d79c9-9deb-4843-b65f-3b0ac8681d62","Type":"ContainerDied","Data":"e85ba2574cbe1d4bfd61ab24aea6322fdb096cef3ad7cfeff2bcb09d87939b06"}
Jul 16 22:24:56.816308 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:56.816216    2555 scope.go:117] "RemoveContainer" containerID="e85ba2574cbe1d4bfd61ab24aea6322fdb096cef3ad7cfeff2bcb09d87939b06"
Jul 16 22:24:57.821312 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:57.821215    2555 generic.go:356] "Generic (PLEG): container finished" podID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerID="4ce0de705c00baf7420f0335f2918218a0244bb30e6b950dfd5f4cbf70a50d57" exitCode=7
Jul 16 22:24:57.821312 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:57.821265    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" event={"ID":"f11d79c9-9deb-4843-b65f-3b0ac8681d62","Type":"ContainerDied","Data":"4ce0de705c00baf7420f0335f2918218a0244bb30e6b950dfd5f4cbf70a50d57"}
Jul 16 22:24:57.821789 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:57.821324    2555 scope.go:117] "RemoveContainer" containerID="e85ba2574cbe1d4bfd61ab24aea6322fdb096cef3ad7cfeff2bcb09d87939b06"
Jul 16 22:24:57.821789 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:57.821525    2555 scope.go:117] "RemoveContainer" containerID="4ce0de705c00baf7420f0335f2918218a0244bb30e6b950dfd5f4cbf70a50d57"
Jul 16 22:24:57.821789 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:57.821760    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25_odh-ai-gateway-infra(f11d79c9-9deb-4843-b65f-3b0ac8681d62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62"
Jul 16 22:24:58.826712 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:24:58.826686    2555 scope.go:117] "RemoveContainer" containerID="4ce0de705c00baf7420f0335f2918218a0244bb30e6b950dfd5f4cbf70a50d57"
Jul 16 22:24:58.827120 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:24:58.826875    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25_odh-ai-gateway-infra(f11d79c9-9deb-4843-b65f-3b0ac8681d62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62"
Jul 16 22:25:02.489795 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:02.489761    2555 scope.go:117] "RemoveContainer" containerID="454a907b66c6ddd452ff6e901ba16e934121a3d598eff5e6797b067d8d77060b"
Jul 16 22:25:03.848718 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:03.848678    2555 generic.go:356] "Generic (PLEG): container finished" podID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerID="9ef32b15c80634ffdb2a7cf7a4d918c041a8a028f280a71d8b97f908002d8d3c" exitCode=6
Jul 16 22:25:03.849139 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:03.848772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" event={"ID":"c70fb956-374c-48b2-b09b-6ad9f0d373b5","Type":"ContainerDied","Data":"9ef32b15c80634ffdb2a7cf7a4d918c041a8a028f280a71d8b97f908002d8d3c"}
Jul 16 22:25:03.849139 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:03.848833    2555 scope.go:117] "RemoveContainer" containerID="454a907b66c6ddd452ff6e901ba16e934121a3d598eff5e6797b067d8d77060b"
Jul 16 22:25:03.849247 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:03.849146    2555 scope.go:117] "RemoveContainer" containerID="9ef32b15c80634ffdb2a7cf7a4d918c041a8a028f280a71d8b97f908002d8d3c"
Jul 16 22:25:03.849357 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:03.849337    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs_odh-ai-gateway-infra(c70fb956-374c-48b2-b09b-6ad9f0d373b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5"
Jul 16 22:25:04.873370 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:04.873339    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"]
Jul 16 22:25:05.004161 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.004137    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"
Jul 16 22:25:05.060121 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.060090    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70fb956-374c-48b2-b09b-6ad9f0d373b5-maas-api-service-ca\") pod \"c70fb956-374c-48b2-b09b-6ad9f0d373b5\" (UID: \"c70fb956-374c-48b2-b09b-6ad9f0d373b5\") "
Jul 16 22:25:05.060489 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.060453    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70fb956-374c-48b2-b09b-6ad9f0d373b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c70fb956-374c-48b2-b09b-6ad9f0d373b5" (UID: "c70fb956-374c-48b2-b09b-6ad9f0d373b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:05.161078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.160984    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c70fb956-374c-48b2-b09b-6ad9f0d373b5-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:25:05.860585 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.860553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs" event={"ID":"c70fb956-374c-48b2-b09b-6ad9f0d373b5","Type":"ContainerDied","Data":"e9dd57f8b2971281ca23a0a9b93813d8c49eec44a980c0dd7846fa5df370fb6d"}
Jul 16 22:25:05.860585 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.860589    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"
Jul 16 22:25:05.860812 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.860612    2555 scope.go:117] "RemoveContainer" containerID="9ef32b15c80634ffdb2a7cf7a4d918c041a8a028f280a71d8b97f908002d8d3c"
Jul 16 22:25:05.881283 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.881245    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"]
Jul 16 22:25:05.889364 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:05.889334    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nqbjs"]
Jul 16 22:25:06.494458 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:06.494428    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" path="/var/lib/kubelet/pods/c70fb956-374c-48b2-b09b-6ad9f0d373b5/volumes"
Jul 16 22:25:10.489983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:10.489925    2555 scope.go:117] "RemoveContainer" containerID="4ce0de705c00baf7420f0335f2918218a0244bb30e6b950dfd5f4cbf70a50d57"
Jul 16 22:25:10.888345 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:10.888310    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" event={"ID":"f11d79c9-9deb-4843-b65f-3b0ac8681d62","Type":"ContainerStarted","Data":"ae1d8c0524744ac4585e16cb328d7b8d9b76bb216db32198e5e057ed516f91bc"}
Jul 16 22:25:10.904886 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:10.904834    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" podStartSLOduration=15.904815554 podStartE2EDuration="15.904815554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:10.904739014 +0000 UTC m=+2266.897903168" watchObservedRunningTime="2026-07-16 22:25:10.904815554 +0000 UTC m=+2266.897979705"
Jul 16 22:25:11.894352 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:11.894322    2555 generic.go:356] "Generic (PLEG): container finished" podID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerID="ae1d8c0524744ac4585e16cb328d7b8d9b76bb216db32198e5e057ed516f91bc" exitCode=7
Jul 16 22:25:11.894884 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:11.894396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" event={"ID":"f11d79c9-9deb-4843-b65f-3b0ac8681d62","Type":"ContainerDied","Data":"ae1d8c0524744ac4585e16cb328d7b8d9b76bb216db32198e5e057ed516f91bc"}
Jul 16 22:25:11.894884 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:11.894458    2555 scope.go:117] "RemoveContainer" containerID="4ce0de705c00baf7420f0335f2918218a0244bb30e6b950dfd5f4cbf70a50d57"
Jul 16 22:25:11.894884 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:11.894737    2555 scope.go:117] "RemoveContainer" containerID="ae1d8c0524744ac4585e16cb328d7b8d9b76bb216db32198e5e057ed516f91bc"
Jul 16 22:25:11.895059 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:11.895023    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25_odh-ai-gateway-infra(f11d79c9-9deb-4843-b65f-3b0ac8681d62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62"
Jul 16 22:25:11.919461 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:11.919431    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"]
Jul 16 22:25:13.032134 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.032102    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"
Jul 16 22:25:13.131970 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.131926    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d79c9-9deb-4843-b65f-3b0ac8681d62-maas-api-service-ca\") pod \"f11d79c9-9deb-4843-b65f-3b0ac8681d62\" (UID: \"f11d79c9-9deb-4843-b65f-3b0ac8681d62\") "
Jul 16 22:25:13.132291 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.132268    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11d79c9-9deb-4843-b65f-3b0ac8681d62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f11d79c9-9deb-4843-b65f-3b0ac8681d62" (UID: "f11d79c9-9deb-4843-b65f-3b0ac8681d62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:13.233018 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.232911    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d79c9-9deb-4843-b65f-3b0ac8681d62-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:25:13.908630 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.908595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25" event={"ID":"f11d79c9-9deb-4843-b65f-3b0ac8681d62","Type":"ContainerDied","Data":"234f8cc4e58e9a5760a977eec78fea8f52750a705426a7b8b8d0f516f467435f"}
Jul 16 22:25:13.908851 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.908646    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"
Jul 16 22:25:13.908851 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.908657    2555 scope.go:117] "RemoveContainer" containerID="ae1d8c0524744ac4585e16cb328d7b8d9b76bb216db32198e5e057ed516f91bc"
Jul 16 22:25:13.929115 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.929089    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"]
Jul 16 22:25:13.937011 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:13.936986    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-8zx25"]
Jul 16 22:25:14.494721 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:14.494687    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" path="/var/lib/kubelet/pods/f11d79c9-9deb-4843-b65f-3b0ac8681d62/volumes"
Jul 16 22:25:24.867607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867572    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"]
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867880    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867890    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867902    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867908    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867917    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867923    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867935    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.867940    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.868011    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.868018    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.868024    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.868030    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:24.868050 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.868037    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:24.872294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.872278    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"
Jul 16 22:25:24.875391 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.875366    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:25:24.875577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.875447    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:25:24.878527 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.878483    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"]
Jul 16 22:25:24.936565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:24.936474    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be094118-1cee-4041-9fc5-6a5ff283320f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4\" (UID: \"be094118-1cee-4041-9fc5-6a5ff283320f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"
Jul 16 22:25:25.037611 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.037574    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be094118-1cee-4041-9fc5-6a5ff283320f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4\" (UID: \"be094118-1cee-4041-9fc5-6a5ff283320f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"
Jul 16 22:25:25.038245 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.038225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be094118-1cee-4041-9fc5-6a5ff283320f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4\" (UID: \"be094118-1cee-4041-9fc5-6a5ff283320f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"
Jul 16 22:25:25.183797 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.183757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"
Jul 16 22:25:25.312380 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.312352    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"]
Jul 16 22:25:25.315162 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:25:25.315131    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe094118_1cee_4041_9fc5_6a5ff283320f.slice/crio-a9835b5355ae1b218169d7c4987a453762f014745eb61afb0b63178ef546e1f7 WatchSource:0}: Error finding container a9835b5355ae1b218169d7c4987a453762f014745eb61afb0b63178ef546e1f7: Status 404 returned error can't find the container with id a9835b5355ae1b218169d7c4987a453762f014745eb61afb0b63178ef546e1f7
Jul 16 22:25:25.962849 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.962819    2555 generic.go:356] "Generic (PLEG): container finished" podID="be094118-1cee-4041-9fc5-6a5ff283320f" containerID="675e5c318f8b655b275acab98ed9ea1229679e5627377670ed44664b305e1981" exitCode=6
Jul 16 22:25:25.963195 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.962878    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" event={"ID":"be094118-1cee-4041-9fc5-6a5ff283320f","Type":"ContainerDied","Data":"675e5c318f8b655b275acab98ed9ea1229679e5627377670ed44664b305e1981"}
Jul 16 22:25:25.963195 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.962928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" event={"ID":"be094118-1cee-4041-9fc5-6a5ff283320f","Type":"ContainerStarted","Data":"a9835b5355ae1b218169d7c4987a453762f014745eb61afb0b63178ef546e1f7"}
Jul 16 22:25:25.963195 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:25.963147    2555 scope.go:117] "RemoveContainer" containerID="675e5c318f8b655b275acab98ed9ea1229679e5627377670ed44664b305e1981"
Jul 16 22:25:26.968784 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:26.968753    2555 generic.go:356] "Generic (PLEG): container finished" podID="be094118-1cee-4041-9fc5-6a5ff283320f" containerID="cecc0c12618a214f419cefec3632783fb4f71a5823b3deda1f520138d6d6c048" exitCode=6
Jul 16 22:25:26.969294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:26.968808    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" event={"ID":"be094118-1cee-4041-9fc5-6a5ff283320f","Type":"ContainerDied","Data":"cecc0c12618a214f419cefec3632783fb4f71a5823b3deda1f520138d6d6c048"}
Jul 16 22:25:26.969294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:26.968872    2555 scope.go:117] "RemoveContainer" containerID="675e5c318f8b655b275acab98ed9ea1229679e5627377670ed44664b305e1981"
Jul 16 22:25:26.969294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:26.969105    2555 scope.go:117] "RemoveContainer" containerID="cecc0c12618a214f419cefec3632783fb4f71a5823b3deda1f520138d6d6c048"
Jul 16 22:25:26.969466 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:26.969343    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4_odh-ai-gateway-infra(be094118-1cee-4041-9fc5-6a5ff283320f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" podUID="be094118-1cee-4041-9fc5-6a5ff283320f"
Jul 16 22:25:27.975445 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:27.975415    2555 scope.go:117] "RemoveContainer" containerID="cecc0c12618a214f419cefec3632783fb4f71a5823b3deda1f520138d6d6c048"
Jul 16 22:25:27.975821 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:27.975626    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4_odh-ai-gateway-infra(be094118-1cee-4041-9fc5-6a5ff283320f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" podUID="be094118-1cee-4041-9fc5-6a5ff283320f"
Jul 16 22:25:36.594247 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.594212    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"]
Jul 16 22:25:36.594656 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.594503    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:36.594656 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.594514    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:36.594656 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.594526    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:36.594656 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.594532    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11d79c9-9deb-4843-b65f-3b0ac8681d62" containerName="revoke-keys"
Jul 16 22:25:36.594656 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.594590    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70fb956-374c-48b2-b09b-6ad9f0d373b5" containerName="revoke-keys"
Jul 16 22:25:36.597492 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.597476    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"
Jul 16 22:25:36.605254 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.605225    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"]
Jul 16 22:25:36.741406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.741368    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51940472-af8d-4be5-af41-90f407ab8bc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm\" (UID: \"51940472-af8d-4be5-af41-90f407ab8bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"
Jul 16 22:25:36.842786 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.842747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51940472-af8d-4be5-af41-90f407ab8bc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm\" (UID: \"51940472-af8d-4be5-af41-90f407ab8bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"
Jul 16 22:25:36.843387 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.843362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51940472-af8d-4be5-af41-90f407ab8bc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm\" (UID: \"51940472-af8d-4be5-af41-90f407ab8bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"
Jul 16 22:25:36.908016 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:36.907928    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"
Jul 16 22:25:37.243914 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:37.243871    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"]
Jul 16 22:25:37.246698 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:25:37.246669    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51940472_af8d_4be5_af41_90f407ab8bc1.slice/crio-2e788a7aa5d380fb917c73b715a5cf63bd6970b4d070fac4dc1094b33b4e38de WatchSource:0}: Error finding container 2e788a7aa5d380fb917c73b715a5cf63bd6970b4d070fac4dc1094b33b4e38de: Status 404 returned error can't find the container with id 2e788a7aa5d380fb917c73b715a5cf63bd6970b4d070fac4dc1094b33b4e38de
Jul 16 22:25:38.015686 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:38.015592    2555 generic.go:356] "Generic (PLEG): container finished" podID="51940472-af8d-4be5-af41-90f407ab8bc1" containerID="c62c879eba10ba67cb8541817b553c83faa55db5183a7bfdb9e6f9d667bb12e8" exitCode=6
Jul 16 22:25:38.016143 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:38.015717    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" event={"ID":"51940472-af8d-4be5-af41-90f407ab8bc1","Type":"ContainerDied","Data":"c62c879eba10ba67cb8541817b553c83faa55db5183a7bfdb9e6f9d667bb12e8"}
Jul 16 22:25:38.016143 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:38.015781    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" event={"ID":"51940472-af8d-4be5-af41-90f407ab8bc1","Type":"ContainerStarted","Data":"2e788a7aa5d380fb917c73b715a5cf63bd6970b4d070fac4dc1094b33b4e38de"}
Jul 16 22:25:38.016143 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:38.015875    2555 scope.go:117] "RemoveContainer" containerID="c62c879eba10ba67cb8541817b553c83faa55db5183a7bfdb9e6f9d667bb12e8"
Jul 16 22:25:39.022044 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:39.021928    2555 generic.go:356] "Generic (PLEG): container finished" podID="51940472-af8d-4be5-af41-90f407ab8bc1" containerID="70a1eb2ce2f627f4e0b8248736f30d1f1ba2794168849a9fcc92a057f52e740c" exitCode=6
Jul 16 22:25:39.022044 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:39.022028    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" event={"ID":"51940472-af8d-4be5-af41-90f407ab8bc1","Type":"ContainerDied","Data":"70a1eb2ce2f627f4e0b8248736f30d1f1ba2794168849a9fcc92a057f52e740c"}
Jul 16 22:25:39.022501 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:39.022091    2555 scope.go:117] "RemoveContainer" containerID="c62c879eba10ba67cb8541817b553c83faa55db5183a7bfdb9e6f9d667bb12e8"
Jul 16 22:25:39.022501 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:39.022311    2555 scope.go:117] "RemoveContainer" containerID="70a1eb2ce2f627f4e0b8248736f30d1f1ba2794168849a9fcc92a057f52e740c"
Jul 16 22:25:39.022567 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:39.022532    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm_odh-ai-gateway-infra(51940472-af8d-4be5-af41-90f407ab8bc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" podUID="51940472-af8d-4be5-af41-90f407ab8bc1"
Jul 16 22:25:40.028579 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:40.028547    2555 scope.go:117] "RemoveContainer" containerID="70a1eb2ce2f627f4e0b8248736f30d1f1ba2794168849a9fcc92a057f52e740c"
Jul 16 22:25:40.028975 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:40.028748    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm_odh-ai-gateway-infra(51940472-af8d-4be5-af41-90f407ab8bc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" podUID="51940472-af8d-4be5-af41-90f407ab8bc1"
Jul 16 22:25:43.489410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:43.489376    2555 scope.go:117] "RemoveContainer" containerID="cecc0c12618a214f419cefec3632783fb4f71a5823b3deda1f520138d6d6c048"
Jul 16 22:25:44.047673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:44.047634    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" event={"ID":"be094118-1cee-4041-9fc5-6a5ff283320f","Type":"ContainerStarted","Data":"b514a3489d2efbb898a27d0c1789744ca21565c62edf9ce7d5cdf0783bc3aafb"}
Jul 16 22:25:44.067854 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:44.067809    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" podStartSLOduration=20.067792242 podStartE2EDuration="20.067792242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:44.067331757 +0000 UTC m=+2300.060495909" watchObservedRunningTime="2026-07-16 22:25:44.067792242 +0000 UTC m=+2300.060956441"
Jul 16 22:25:45.054110 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:45.054070    2555 generic.go:356] "Generic (PLEG): container finished" podID="be094118-1cee-4041-9fc5-6a5ff283320f" containerID="b514a3489d2efbb898a27d0c1789744ca21565c62edf9ce7d5cdf0783bc3aafb" exitCode=6
Jul 16 22:25:45.054540 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:45.054121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" event={"ID":"be094118-1cee-4041-9fc5-6a5ff283320f","Type":"ContainerDied","Data":"b514a3489d2efbb898a27d0c1789744ca21565c62edf9ce7d5cdf0783bc3aafb"}
Jul 16 22:25:45.054540 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:45.054195    2555 scope.go:117] "RemoveContainer" containerID="cecc0c12618a214f419cefec3632783fb4f71a5823b3deda1f520138d6d6c048"
Jul 16 22:25:45.054540 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:45.054494    2555 scope.go:117] "RemoveContainer" containerID="b514a3489d2efbb898a27d0c1789744ca21565c62edf9ce7d5cdf0783bc3aafb"
Jul 16 22:25:45.054712 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:45.054680    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4_odh-ai-gateway-infra(be094118-1cee-4041-9fc5-6a5ff283320f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" podUID="be094118-1cee-4041-9fc5-6a5ff283320f"
Jul 16 22:25:45.083368 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:45.083335    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"]
Jul 16 22:25:46.195109 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:46.195082    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"
Jul 16 22:25:46.319601 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:46.319514    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be094118-1cee-4041-9fc5-6a5ff283320f-maas-api-service-ca\") pod \"be094118-1cee-4041-9fc5-6a5ff283320f\" (UID: \"be094118-1cee-4041-9fc5-6a5ff283320f\") "
Jul 16 22:25:46.319884 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:46.319862    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be094118-1cee-4041-9fc5-6a5ff283320f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be094118-1cee-4041-9fc5-6a5ff283320f" (UID: "be094118-1cee-4041-9fc5-6a5ff283320f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:46.420333 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:46.420295    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be094118-1cee-4041-9fc5-6a5ff283320f-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:25:47.064874 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:47.064835    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4" event={"ID":"be094118-1cee-4041-9fc5-6a5ff283320f","Type":"ContainerDied","Data":"a9835b5355ae1b218169d7c4987a453762f014745eb61afb0b63178ef546e1f7"}
Jul 16 22:25:47.064874 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:47.064870    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"
Jul 16 22:25:47.065152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:47.064907    2555 scope.go:117] "RemoveContainer" containerID="b514a3489d2efbb898a27d0c1789744ca21565c62edf9ce7d5cdf0783bc3aafb"
Jul 16 22:25:47.089282 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:47.089249    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"]
Jul 16 22:25:47.095491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:47.095468    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ddfk4"]
Jul 16 22:25:48.495387 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:48.495352    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" path="/var/lib/kubelet/pods/be094118-1cee-4041-9fc5-6a5ff283320f/volumes"
Jul 16 22:25:54.491838 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:54.491803    2555 scope.go:117] "RemoveContainer" containerID="70a1eb2ce2f627f4e0b8248736f30d1f1ba2794168849a9fcc92a057f52e740c"
Jul 16 22:25:55.100537 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:55.100504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" event={"ID":"51940472-af8d-4be5-af41-90f407ab8bc1","Type":"ContainerStarted","Data":"93ed9eadb045856a7610ecea5a15e50ff8b2141029244541835ec696df623b7a"}
Jul 16 22:25:55.117149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:55.117100    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" podStartSLOduration=19.117084392 podStartE2EDuration="19.117084392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:55.115477949 +0000 UTC m=+2311.108642102" watchObservedRunningTime="2026-07-16 22:25:55.117084392 +0000 UTC m=+2311.110248573"
Jul 16 22:25:56.106872 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:56.106840    2555 generic.go:356] "Generic (PLEG): container finished" podID="51940472-af8d-4be5-af41-90f407ab8bc1" containerID="93ed9eadb045856a7610ecea5a15e50ff8b2141029244541835ec696df623b7a" exitCode=6
Jul 16 22:25:56.107296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:56.106879    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" event={"ID":"51940472-af8d-4be5-af41-90f407ab8bc1","Type":"ContainerDied","Data":"93ed9eadb045856a7610ecea5a15e50ff8b2141029244541835ec696df623b7a"}
Jul 16 22:25:56.107296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:56.106933    2555 scope.go:117] "RemoveContainer" containerID="70a1eb2ce2f627f4e0b8248736f30d1f1ba2794168849a9fcc92a057f52e740c"
Jul 16 22:25:56.107296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:56.107247    2555 scope.go:117] "RemoveContainer" containerID="93ed9eadb045856a7610ecea5a15e50ff8b2141029244541835ec696df623b7a"
Jul 16 22:25:56.107460 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:25:56.107439    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm_odh-ai-gateway-infra(51940472-af8d-4be5-af41-90f407ab8bc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" podUID="51940472-af8d-4be5-af41-90f407ab8bc1"
Jul 16 22:25:56.126128 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:56.126106    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"]
Jul 16 22:25:57.244021 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:57.243994    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"
Jul 16 22:25:57.312025 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:57.311983    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51940472-af8d-4be5-af41-90f407ab8bc1-maas-api-service-ca\") pod \"51940472-af8d-4be5-af41-90f407ab8bc1\" (UID: \"51940472-af8d-4be5-af41-90f407ab8bc1\") "
Jul 16 22:25:57.312340 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:57.312317    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51940472-af8d-4be5-af41-90f407ab8bc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51940472-af8d-4be5-af41-90f407ab8bc1" (UID: "51940472-af8d-4be5-af41-90f407ab8bc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:57.413586 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:57.413505    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51940472-af8d-4be5-af41-90f407ab8bc1-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:25:58.118437 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:58.118406    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"
Jul 16 22:25:58.118644 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:58.118404    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm" event={"ID":"51940472-af8d-4be5-af41-90f407ab8bc1","Type":"ContainerDied","Data":"2e788a7aa5d380fb917c73b715a5cf63bd6970b4d070fac4dc1094b33b4e38de"}
Jul 16 22:25:58.118644 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:58.118547    2555 scope.go:117] "RemoveContainer" containerID="93ed9eadb045856a7610ecea5a15e50ff8b2141029244541835ec696df623b7a"
Jul 16 22:25:58.143209 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:58.143181    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"]
Jul 16 22:25:58.145734 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:58.145690    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtjdm"]
Jul 16 22:25:58.495168 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:25:58.495069    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" path="/var/lib/kubelet/pods/51940472-af8d-4be5-af41-90f407ab8bc1/volumes"
Jul 16 22:26:24.924082 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924001    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"]
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924296    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924306    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924319    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924324    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924343    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924349    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924357    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924363    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924369    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924374    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924419    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924432    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924439    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:24.924463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.924443    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:24.929182 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.929160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"
Jul 16 22:26:24.931913 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.931891    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:26:24.932041 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.931951    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:26:24.935371 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:24.935343    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"]
Jul 16 22:26:25.061020 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:25.060978    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0402eb2-d55f-40b2-9e4e-021fc17ffde4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-6jb22\" (UID: \"f0402eb2-d55f-40b2-9e4e-021fc17ffde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"
Jul 16 22:26:25.162003 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:25.161944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0402eb2-d55f-40b2-9e4e-021fc17ffde4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-6jb22\" (UID: \"f0402eb2-d55f-40b2-9e4e-021fc17ffde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"
Jul 16 22:26:25.162564 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:25.162544    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0402eb2-d55f-40b2-9e4e-021fc17ffde4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-6jb22\" (UID: \"f0402eb2-d55f-40b2-9e4e-021fc17ffde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"
Jul 16 22:26:25.241238 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:25.241158    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"
Jul 16 22:26:25.377645 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:25.377613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"]
Jul 16 22:26:25.380809 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:26:25.380781    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0402eb2_d55f_40b2_9e4e_021fc17ffde4.slice/crio-eba1c0ab8c4d798131bb270d7e18bd99b8310ade48594da766229c0bbab2a71a WatchSource:0}: Error finding container eba1c0ab8c4d798131bb270d7e18bd99b8310ade48594da766229c0bbab2a71a: Status 404 returned error can't find the container with id eba1c0ab8c4d798131bb270d7e18bd99b8310ade48594da766229c0bbab2a71a
Jul 16 22:26:26.237650 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:26.237613    2555 generic.go:356] "Generic (PLEG): container finished" podID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerID="468edd62d6ec31431b288a56cc2bd20cbaf8feec288215a0ace84a5baeeac3c8" exitCode=6
Jul 16 22:26:26.238058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:26.237673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" event={"ID":"f0402eb2-d55f-40b2-9e4e-021fc17ffde4","Type":"ContainerDied","Data":"468edd62d6ec31431b288a56cc2bd20cbaf8feec288215a0ace84a5baeeac3c8"}
Jul 16 22:26:26.238058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:26.237720    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" event={"ID":"f0402eb2-d55f-40b2-9e4e-021fc17ffde4","Type":"ContainerStarted","Data":"eba1c0ab8c4d798131bb270d7e18bd99b8310ade48594da766229c0bbab2a71a"}
Jul 16 22:26:26.238058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:26.237908    2555 scope.go:117] "RemoveContainer" containerID="468edd62d6ec31431b288a56cc2bd20cbaf8feec288215a0ace84a5baeeac3c8"
Jul 16 22:26:27.244147 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.244057    2555 generic.go:356] "Generic (PLEG): container finished" podID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerID="f23a530944066e74d5711b6c12f572602282482cda89c3c85e5cb42bedd2a419" exitCode=6
Jul 16 22:26:27.244147 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.244107    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" event={"ID":"f0402eb2-d55f-40b2-9e4e-021fc17ffde4","Type":"ContainerDied","Data":"f23a530944066e74d5711b6c12f572602282482cda89c3c85e5cb42bedd2a419"}
Jul 16 22:26:27.244617 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.244167    2555 scope.go:117] "RemoveContainer" containerID="468edd62d6ec31431b288a56cc2bd20cbaf8feec288215a0ace84a5baeeac3c8"
Jul 16 22:26:27.244617 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.244332    2555 scope.go:117] "RemoveContainer" containerID="f23a530944066e74d5711b6c12f572602282482cda89c3c85e5cb42bedd2a419"
Jul 16 22:26:27.244617 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:27.244578    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-6jb22_odh-ai-gateway-infra(f0402eb2-d55f-40b2-9e4e-021fc17ffde4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4"
Jul 16 22:26:27.947549 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.947520    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"]
Jul 16 22:26:27.947837 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.947810    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:27.947837 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.947826    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="51940472-af8d-4be5-af41-90f407ab8bc1" containerName="revoke-keys"
Jul 16 22:26:27.948043 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.947917    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="be094118-1cee-4041-9fc5-6a5ff283320f" containerName="revoke-keys"
Jul 16 22:26:27.952100 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.952077    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"
Jul 16 22:26:27.962985 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:27.962927    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"]
Jul 16 22:26:28.100007 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:28.099973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82c00a5-106f-4e06-8d3e-a25d493fa0d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v\" (UID: \"c82c00a5-106f-4e06-8d3e-a25d493fa0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"
Jul 16 22:26:28.200530 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:28.200448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82c00a5-106f-4e06-8d3e-a25d493fa0d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v\" (UID: \"c82c00a5-106f-4e06-8d3e-a25d493fa0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"
Jul 16 22:26:28.201090 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:28.201070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82c00a5-106f-4e06-8d3e-a25d493fa0d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v\" (UID: \"c82c00a5-106f-4e06-8d3e-a25d493fa0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"
Jul 16 22:26:28.249571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:28.249548    2555 scope.go:117] "RemoveContainer" containerID="f23a530944066e74d5711b6c12f572602282482cda89c3c85e5cb42bedd2a419"
Jul 16 22:26:28.249981 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:28.249735    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-6jb22_odh-ai-gateway-infra(f0402eb2-d55f-40b2-9e4e-021fc17ffde4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4"
Jul 16 22:26:28.262507 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:28.262485    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"
Jul 16 22:26:28.626582 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:28.626545    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"]
Jul 16 22:26:28.630188 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:26:28.630158    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82c00a5_106f_4e06_8d3e_a25d493fa0d9.slice/crio-73da4865d10a773676f0a59b7af0eedeb8cde1d9299aaa99e0c4aa4fd7f5eab4 WatchSource:0}: Error finding container 73da4865d10a773676f0a59b7af0eedeb8cde1d9299aaa99e0c4aa4fd7f5eab4: Status 404 returned error can't find the container with id 73da4865d10a773676f0a59b7af0eedeb8cde1d9299aaa99e0c4aa4fd7f5eab4
Jul 16 22:26:29.254750 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:29.254669    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" event={"ID":"c82c00a5-106f-4e06-8d3e-a25d493fa0d9","Type":"ContainerStarted","Data":"17d90fef49a83cec5d0ca053f95a46c14f260b6724a6db18817f89b5059f2fdc"}
Jul 16 22:26:29.254750 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:29.254718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" event={"ID":"c82c00a5-106f-4e06-8d3e-a25d493fa0d9","Type":"ContainerStarted","Data":"73da4865d10a773676f0a59b7af0eedeb8cde1d9299aaa99e0c4aa4fd7f5eab4"}
Jul 16 22:26:29.278000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:29.277937    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" podStartSLOduration=2.277921757 podStartE2EDuration="2.277921757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:29.275809448 +0000 UTC m=+2345.268973601" watchObservedRunningTime="2026-07-16 22:26:29.277921757 +0000 UTC m=+2345.271085909"
Jul 16 22:26:30.260312 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:30.260278    2555 generic.go:356] "Generic (PLEG): container finished" podID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerID="17d90fef49a83cec5d0ca053f95a46c14f260b6724a6db18817f89b5059f2fdc" exitCode=7
Jul 16 22:26:30.260762 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:30.260365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" event={"ID":"c82c00a5-106f-4e06-8d3e-a25d493fa0d9","Type":"ContainerDied","Data":"17d90fef49a83cec5d0ca053f95a46c14f260b6724a6db18817f89b5059f2fdc"}
Jul 16 22:26:30.260762 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:30.260567    2555 scope.go:117] "RemoveContainer" containerID="17d90fef49a83cec5d0ca053f95a46c14f260b6724a6db18817f89b5059f2fdc"
Jul 16 22:26:31.265727 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:31.265629    2555 generic.go:356] "Generic (PLEG): container finished" podID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerID="3bec16b487180150c91af9d01b09574a3b5c955e4677f7852e472820f06d6479" exitCode=7
Jul 16 22:26:31.265727 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:31.265698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" event={"ID":"c82c00a5-106f-4e06-8d3e-a25d493fa0d9","Type":"ContainerDied","Data":"3bec16b487180150c91af9d01b09574a3b5c955e4677f7852e472820f06d6479"}
Jul 16 22:26:31.266196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:31.265749    2555 scope.go:117] "RemoveContainer" containerID="17d90fef49a83cec5d0ca053f95a46c14f260b6724a6db18817f89b5059f2fdc"
Jul 16 22:26:31.266196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:31.265922    2555 scope.go:117] "RemoveContainer" containerID="3bec16b487180150c91af9d01b09574a3b5c955e4677f7852e472820f06d6479"
Jul 16 22:26:31.266196 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:31.266177    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v_odh-ai-gateway-infra(c82c00a5-106f-4e06-8d3e-a25d493fa0d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9"
Jul 16 22:26:32.271104 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:32.271075    2555 scope.go:117] "RemoveContainer" containerID="3bec16b487180150c91af9d01b09574a3b5c955e4677f7852e472820f06d6479"
Jul 16 22:26:32.271498 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:32.271282    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v_odh-ai-gateway-infra(c82c00a5-106f-4e06-8d3e-a25d493fa0d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9"
Jul 16 22:26:39.490171 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:39.490140    2555 scope.go:117] "RemoveContainer" containerID="f23a530944066e74d5711b6c12f572602282482cda89c3c85e5cb42bedd2a419"
Jul 16 22:26:40.305285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:40.305191    2555 generic.go:356] "Generic (PLEG): container finished" podID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerID="687097c2606601c9bcf0749e1d6391e78ef3b02c024e9f790fb33f58603aaa63" exitCode=6
Jul 16 22:26:40.305285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:40.305257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" event={"ID":"f0402eb2-d55f-40b2-9e4e-021fc17ffde4","Type":"ContainerDied","Data":"687097c2606601c9bcf0749e1d6391e78ef3b02c024e9f790fb33f58603aaa63"}
Jul 16 22:26:40.305511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:40.305331    2555 scope.go:117] "RemoveContainer" containerID="f23a530944066e74d5711b6c12f572602282482cda89c3c85e5cb42bedd2a419"
Jul 16 22:26:40.305699 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:40.305681    2555 scope.go:117] "RemoveContainer" containerID="687097c2606601c9bcf0749e1d6391e78ef3b02c024e9f790fb33f58603aaa63"
Jul 16 22:26:40.305926 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:40.305903    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-6jb22_odh-ai-gateway-infra(f0402eb2-d55f-40b2-9e4e-021fc17ffde4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4"
Jul 16 22:26:41.339510 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:41.339475    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"]
Jul 16 22:26:41.477072 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:41.477044    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"
Jul 16 22:26:41.518129 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:41.518099    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0402eb2-d55f-40b2-9e4e-021fc17ffde4-maas-api-service-ca\") pod \"f0402eb2-d55f-40b2-9e4e-021fc17ffde4\" (UID: \"f0402eb2-d55f-40b2-9e4e-021fc17ffde4\") "
Jul 16 22:26:41.518464 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:41.518437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0402eb2-d55f-40b2-9e4e-021fc17ffde4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0402eb2-d55f-40b2-9e4e-021fc17ffde4" (UID: "f0402eb2-d55f-40b2-9e4e-021fc17ffde4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:41.619619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:41.619534    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0402eb2-d55f-40b2-9e4e-021fc17ffde4-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:26:42.322438 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:42.322404    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"
Jul 16 22:26:42.322438 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:42.322423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22" event={"ID":"f0402eb2-d55f-40b2-9e4e-021fc17ffde4","Type":"ContainerDied","Data":"eba1c0ab8c4d798131bb270d7e18bd99b8310ade48594da766229c0bbab2a71a"}
Jul 16 22:26:42.322697 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:42.322520    2555 scope.go:117] "RemoveContainer" containerID="687097c2606601c9bcf0749e1d6391e78ef3b02c024e9f790fb33f58603aaa63"
Jul 16 22:26:42.345511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:42.345474    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"]
Jul 16 22:26:42.353519 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:42.353495    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-6jb22"]
Jul 16 22:26:42.494308 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:42.494273    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" path="/var/lib/kubelet/pods/f0402eb2-d55f-40b2-9e4e-021fc17ffde4/volumes"
Jul 16 22:26:45.299427 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299392    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"]
Jul 16 22:26:45.299815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299718    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.299815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299728    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.299815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299738    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.299815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299744    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.299815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299759    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.299815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299765    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.299815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299815    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.300063 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.299824    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:26:45.304031 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.304013    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"
Jul 16 22:26:45.321110 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.321080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"]
Jul 16 22:26:45.352346 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.352314    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd706c1f-d9f9-4041-af1b-1662368a9760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km\" (UID: \"bd706c1f-d9f9-4041-af1b-1662368a9760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"
Jul 16 22:26:45.453785 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.453744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd706c1f-d9f9-4041-af1b-1662368a9760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km\" (UID: \"bd706c1f-d9f9-4041-af1b-1662368a9760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"
Jul 16 22:26:45.454406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.454385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd706c1f-d9f9-4041-af1b-1662368a9760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km\" (UID: \"bd706c1f-d9f9-4041-af1b-1662368a9760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"
Jul 16 22:26:45.490158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.490131    2555 scope.go:117] "RemoveContainer" containerID="3bec16b487180150c91af9d01b09574a3b5c955e4677f7852e472820f06d6479"
Jul 16 22:26:45.613903 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.613862    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"
Jul 16 22:26:45.831098 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:45.831068    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"]
Jul 16 22:26:45.832115 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:26:45.832094    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd706c1f_d9f9_4041_af1b_1662368a9760.slice/crio-59357dacc2a16b1541bc4a9fa0850bbe5f3024c54986118c638332ab4b4dbb04 WatchSource:0}: Error finding container 59357dacc2a16b1541bc4a9fa0850bbe5f3024c54986118c638332ab4b4dbb04: Status 404 returned error can't find the container with id 59357dacc2a16b1541bc4a9fa0850bbe5f3024c54986118c638332ab4b4dbb04
Jul 16 22:26:46.342605 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:46.342572    2555 generic.go:356] "Generic (PLEG): container finished" podID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerID="b67464b36a78b71ac4e0ff69ec9e839ca89c0b32468e38fd128fb5e6d4adfbe2" exitCode=7
Jul 16 22:26:46.343055 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:46.342647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" event={"ID":"c82c00a5-106f-4e06-8d3e-a25d493fa0d9","Type":"ContainerDied","Data":"b67464b36a78b71ac4e0ff69ec9e839ca89c0b32468e38fd128fb5e6d4adfbe2"}
Jul 16 22:26:46.343055 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:46.342711    2555 scope.go:117] "RemoveContainer" containerID="3bec16b487180150c91af9d01b09574a3b5c955e4677f7852e472820f06d6479"
Jul 16 22:26:46.343055 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:46.343050    2555 scope.go:117] "RemoveContainer" containerID="b67464b36a78b71ac4e0ff69ec9e839ca89c0b32468e38fd128fb5e6d4adfbe2"
Jul 16 22:26:46.343322 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:46.343288    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v_odh-ai-gateway-infra(c82c00a5-106f-4e06-8d3e-a25d493fa0d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9"
Jul 16 22:26:46.344381 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:46.344358    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" event={"ID":"bd706c1f-d9f9-4041-af1b-1662368a9760","Type":"ContainerStarted","Data":"328c659760b087a2a783921fe2125e5a066f9a9866d48e279eb1fd9b62310da8"}
Jul 16 22:26:46.344469 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:46.344411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" event={"ID":"bd706c1f-d9f9-4041-af1b-1662368a9760","Type":"ContainerStarted","Data":"59357dacc2a16b1541bc4a9fa0850bbe5f3024c54986118c638332ab4b4dbb04"}
Jul 16 22:26:46.380791 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:46.380735    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" podStartSLOduration=1.380718297 podStartE2EDuration="1.380718297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:46.378536799 +0000 UTC m=+2362.371700953" watchObservedRunningTime="2026-07-16 22:26:46.380718297 +0000 UTC m=+2362.373882449"
Jul 16 22:26:47.351930 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.351895    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerID="328c659760b087a2a783921fe2125e5a066f9a9866d48e279eb1fd9b62310da8" exitCode=6
Jul 16 22:26:47.352335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.351991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" event={"ID":"bd706c1f-d9f9-4041-af1b-1662368a9760","Type":"ContainerDied","Data":"328c659760b087a2a783921fe2125e5a066f9a9866d48e279eb1fd9b62310da8"}
Jul 16 22:26:47.352335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.352181    2555 scope.go:117] "RemoveContainer" containerID="328c659760b087a2a783921fe2125e5a066f9a9866d48e279eb1fd9b62310da8"
Jul 16 22:26:47.382081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.382056    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"]
Jul 16 22:26:47.516296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.516269    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"
Jul 16 22:26:47.573135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.573099    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82c00a5-106f-4e06-8d3e-a25d493fa0d9-maas-api-service-ca\") pod \"c82c00a5-106f-4e06-8d3e-a25d493fa0d9\" (UID: \"c82c00a5-106f-4e06-8d3e-a25d493fa0d9\") "
Jul 16 22:26:47.573482 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.573459    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82c00a5-106f-4e06-8d3e-a25d493fa0d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c82c00a5-106f-4e06-8d3e-a25d493fa0d9" (UID: "c82c00a5-106f-4e06-8d3e-a25d493fa0d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:47.674745 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:47.674699    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82c00a5-106f-4e06-8d3e-a25d493fa0d9-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:26:48.358361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.358331    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"
Jul 16 22:26:48.358801 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.358333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v" event={"ID":"c82c00a5-106f-4e06-8d3e-a25d493fa0d9","Type":"ContainerDied","Data":"73da4865d10a773676f0a59b7af0eedeb8cde1d9299aaa99e0c4aa4fd7f5eab4"}
Jul 16 22:26:48.358801 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.358466    2555 scope.go:117] "RemoveContainer" containerID="b67464b36a78b71ac4e0ff69ec9e839ca89c0b32468e38fd128fb5e6d4adfbe2"
Jul 16 22:26:48.360341 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.360315    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerID="085122db6986c4d90ca6b6dbdfbde4bfa1212715679fcb946b6761d954449dc8" exitCode=6
Jul 16 22:26:48.360459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.360401    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" event={"ID":"bd706c1f-d9f9-4041-af1b-1662368a9760","Type":"ContainerDied","Data":"085122db6986c4d90ca6b6dbdfbde4bfa1212715679fcb946b6761d954449dc8"}
Jul 16 22:26:48.360587 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.360573    2555 scope.go:117] "RemoveContainer" containerID="085122db6986c4d90ca6b6dbdfbde4bfa1212715679fcb946b6761d954449dc8"
Jul 16 22:26:48.360751 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:48.360736    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km_odh-ai-gateway-infra(bd706c1f-d9f9-4041-af1b-1662368a9760)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760"
Jul 16 22:26:48.366512 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.366493    2555 scope.go:117] "RemoveContainer" containerID="328c659760b087a2a783921fe2125e5a066f9a9866d48e279eb1fd9b62310da8"
Jul 16 22:26:48.430584 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.430552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"]
Jul 16 22:26:48.438712 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.438683    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-dxh8v"]
Jul 16 22:26:48.495434 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:48.495408    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" path="/var/lib/kubelet/pods/c82c00a5-106f-4e06-8d3e-a25d493fa0d9/volumes"
Jul 16 22:26:49.366508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:26:49.366477    2555 scope.go:117] "RemoveContainer" containerID="085122db6986c4d90ca6b6dbdfbde4bfa1212715679fcb946b6761d954449dc8"
Jul 16 22:26:49.366879 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:26:49.366661    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km_odh-ai-gateway-infra(bd706c1f-d9f9-4041-af1b-1662368a9760)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760"
Jul 16 22:27:01.490370 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:01.490340    2555 scope.go:117] "RemoveContainer" containerID="085122db6986c4d90ca6b6dbdfbde4bfa1212715679fcb946b6761d954449dc8"
Jul 16 22:27:02.421907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:02.421875    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerID="87056c1cc281ffed43cdc8cbde1d0539fad313e4f44e6d5905b9116d67ddc739" exitCode=6
Jul 16 22:27:02.422106 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:02.421917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" event={"ID":"bd706c1f-d9f9-4041-af1b-1662368a9760","Type":"ContainerDied","Data":"87056c1cc281ffed43cdc8cbde1d0539fad313e4f44e6d5905b9116d67ddc739"}
Jul 16 22:27:02.422106 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:02.421991    2555 scope.go:117] "RemoveContainer" containerID="085122db6986c4d90ca6b6dbdfbde4bfa1212715679fcb946b6761d954449dc8"
Jul 16 22:27:02.422338 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:02.422322    2555 scope.go:117] "RemoveContainer" containerID="87056c1cc281ffed43cdc8cbde1d0539fad313e4f44e6d5905b9116d67ddc739"
Jul 16 22:27:02.422587 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:02.422563    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km_odh-ai-gateway-infra(bd706c1f-d9f9-4041-af1b-1662368a9760)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760"
Jul 16 22:27:03.455338 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:03.455299    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"]
Jul 16 22:27:03.584187 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:03.584157    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"
Jul 16 22:27:03.712429 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:03.712334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd706c1f-d9f9-4041-af1b-1662368a9760-maas-api-service-ca\") pod \"bd706c1f-d9f9-4041-af1b-1662368a9760\" (UID: \"bd706c1f-d9f9-4041-af1b-1662368a9760\") "
Jul 16 22:27:03.712729 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:03.712704    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd706c1f-d9f9-4041-af1b-1662368a9760-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd706c1f-d9f9-4041-af1b-1662368a9760" (UID: "bd706c1f-d9f9-4041-af1b-1662368a9760"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:03.813459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:03.813420    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd706c1f-d9f9-4041-af1b-1662368a9760-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:27:04.433416 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:04.433380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km" event={"ID":"bd706c1f-d9f9-4041-af1b-1662368a9760","Type":"ContainerDied","Data":"59357dacc2a16b1541bc4a9fa0850bbe5f3024c54986118c638332ab4b4dbb04"}
Jul 16 22:27:04.433416 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:04.433402    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"
Jul 16 22:27:04.433660 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:04.433467    2555 scope.go:117] "RemoveContainer" containerID="87056c1cc281ffed43cdc8cbde1d0539fad313e4f44e6d5905b9116d67ddc739"
Jul 16 22:27:04.455333 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:04.455304    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"]
Jul 16 22:27:04.461578 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:04.461553    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-gs9km"]
Jul 16 22:27:04.495685 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:04.495652    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" path="/var/lib/kubelet/pods/bd706c1f-d9f9-4041-af1b-1662368a9760/volumes"
Jul 16 22:27:14.973761 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.973725    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"]
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974029    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974041    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974051    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974058    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974065    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974070    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974087    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974093    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974138    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974144    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974153    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974158    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974165    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0402eb2-d55f-40b2-9e4e-021fc17ffde4" containerName="revoke-keys"
Jul 16 22:27:14.974248 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.974170    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:14.978422 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.978403    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"
Jul 16 22:27:14.981167 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.981146    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:27:14.981167 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.981151    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:27:14.984726 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:14.984698    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"]
Jul 16 22:27:15.003865 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:15.003832    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6bc68-d3f1-4fe6-83fd-e4148a670808-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s\" (UID: \"83e6bc68-d3f1-4fe6-83fd-e4148a670808\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"
Jul 16 22:27:15.104382 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:15.104339    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6bc68-d3f1-4fe6-83fd-e4148a670808-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s\" (UID: \"83e6bc68-d3f1-4fe6-83fd-e4148a670808\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"
Jul 16 22:27:15.105029 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:15.104998    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6bc68-d3f1-4fe6-83fd-e4148a670808-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s\" (UID: \"83e6bc68-d3f1-4fe6-83fd-e4148a670808\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"
Jul 16 22:27:15.289992 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:15.289882    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"
Jul 16 22:27:15.436862 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:15.436834    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"]
Jul 16 22:27:15.439280 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:27:15.439253    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e6bc68_d3f1_4fe6_83fd_e4148a670808.slice/crio-95bfe3f47652f7a33c09e78eca2c5db254cef9ad44797e0b8e45b1c011704647 WatchSource:0}: Error finding container 95bfe3f47652f7a33c09e78eca2c5db254cef9ad44797e0b8e45b1c011704647: Status 404 returned error can't find the container with id 95bfe3f47652f7a33c09e78eca2c5db254cef9ad44797e0b8e45b1c011704647
Jul 16 22:27:15.483259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:15.483227    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" event={"ID":"83e6bc68-d3f1-4fe6-83fd-e4148a670808","Type":"ContainerStarted","Data":"95bfe3f47652f7a33c09e78eca2c5db254cef9ad44797e0b8e45b1c011704647"}
Jul 16 22:27:16.488632 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:16.488600    2555 generic.go:356] "Generic (PLEG): container finished" podID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerID="077fbc0e660f17c0a3dde0259117e3e288755ea7f272d22aca5b495c6e7b9a4c" exitCode=6
Jul 16 22:27:16.489080 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:16.488683    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" event={"ID":"83e6bc68-d3f1-4fe6-83fd-e4148a670808","Type":"ContainerDied","Data":"077fbc0e660f17c0a3dde0259117e3e288755ea7f272d22aca5b495c6e7b9a4c"}
Jul 16 22:27:16.489080 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:16.488840    2555 scope.go:117] "RemoveContainer" containerID="077fbc0e660f17c0a3dde0259117e3e288755ea7f272d22aca5b495c6e7b9a4c"
Jul 16 22:27:17.495173 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:17.495141    2555 generic.go:356] "Generic (PLEG): container finished" podID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerID="4b222c2de47cbf3466e0c4276fcc7e8d1f1c3447a555fba5351ff4d2f6f4d864" exitCode=6
Jul 16 22:27:17.495673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:17.495195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" event={"ID":"83e6bc68-d3f1-4fe6-83fd-e4148a670808","Type":"ContainerDied","Data":"4b222c2de47cbf3466e0c4276fcc7e8d1f1c3447a555fba5351ff4d2f6f4d864"}
Jul 16 22:27:17.495673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:17.495242    2555 scope.go:117] "RemoveContainer" containerID="077fbc0e660f17c0a3dde0259117e3e288755ea7f272d22aca5b495c6e7b9a4c"
Jul 16 22:27:17.495673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:17.495426    2555 scope.go:117] "RemoveContainer" containerID="4b222c2de47cbf3466e0c4276fcc7e8d1f1c3447a555fba5351ff4d2f6f4d864"
Jul 16 22:27:17.495673 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:17.495637    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s_odh-ai-gateway-infra(83e6bc68-d3f1-4fe6-83fd-e4148a670808)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808"
Jul 16 22:27:18.500064 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:18.500033    2555 scope.go:117] "RemoveContainer" containerID="4b222c2de47cbf3466e0c4276fcc7e8d1f1c3447a555fba5351ff4d2f6f4d864"
Jul 16 22:27:18.500458 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:18.500217    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s_odh-ai-gateway-infra(83e6bc68-d3f1-4fe6-83fd-e4148a670808)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808"
Jul 16 22:27:27.993890 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:27.993855    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"]
Jul 16 22:27:27.994295 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:27.994165    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:27.994295 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:27.994177    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:27.994295 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:27.994186    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:27.994295 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:27.994192    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd706c1f-d9f9-4041-af1b-1662368a9760" containerName="revoke-keys"
Jul 16 22:27:27.994295 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:27.994254    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82c00a5-106f-4e06-8d3e-a25d493fa0d9" containerName="revoke-keys"
Jul 16 22:27:27.998545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:27.998524    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"
Jul 16 22:27:28.004705 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:28.004680    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"]
Jul 16 22:27:28.119502 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:28.119466    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-92snh\" (UID: \"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"
Jul 16 22:27:28.220307 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:28.220262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-92snh\" (UID: \"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"
Jul 16 22:27:28.220906 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:28.220885    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-92snh\" (UID: \"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"
Jul 16 22:27:28.309392 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:28.309302    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"
Jul 16 22:27:28.439639 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:28.439462    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"]
Jul 16 22:27:28.442175 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:27:28.442142    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fcd66e_9fa5_4879_9ba4_96ed9ba1f2fc.slice/crio-021e58d6c21abc4406b584218c364667ab70d339e8557a7fbea6c15553ca155d WatchSource:0}: Error finding container 021e58d6c21abc4406b584218c364667ab70d339e8557a7fbea6c15553ca155d: Status 404 returned error can't find the container with id 021e58d6c21abc4406b584218c364667ab70d339e8557a7fbea6c15553ca155d
Jul 16 22:27:28.540237 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:28.540202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" event={"ID":"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc","Type":"ContainerStarted","Data":"021e58d6c21abc4406b584218c364667ab70d339e8557a7fbea6c15553ca155d"}
Jul 16 22:27:29.545950 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:29.545909    2555 generic.go:356] "Generic (PLEG): container finished" podID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerID="6fe9b4de5660679ffce288d17912c400e48534452633b93761fd435cc9e0a5f9" exitCode=7
Jul 16 22:27:29.546379 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:29.546004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" event={"ID":"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc","Type":"ContainerDied","Data":"6fe9b4de5660679ffce288d17912c400e48534452633b93761fd435cc9e0a5f9"}
Jul 16 22:27:29.546379 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:29.546202    2555 scope.go:117] "RemoveContainer" containerID="6fe9b4de5660679ffce288d17912c400e48534452633b93761fd435cc9e0a5f9"
Jul 16 22:27:30.551892 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:30.551793    2555 generic.go:356] "Generic (PLEG): container finished" podID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerID="6919fccd468b878585b16290e8d2599e1b86f6a77fd1131bc678788c99a8a311" exitCode=7
Jul 16 22:27:30.551892 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:30.551833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" event={"ID":"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc","Type":"ContainerDied","Data":"6919fccd468b878585b16290e8d2599e1b86f6a77fd1131bc678788c99a8a311"}
Jul 16 22:27:30.551892 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:30.551891    2555 scope.go:117] "RemoveContainer" containerID="6fe9b4de5660679ffce288d17912c400e48534452633b93761fd435cc9e0a5f9"
Jul 16 22:27:30.552393 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:30.552120    2555 scope.go:117] "RemoveContainer" containerID="6919fccd468b878585b16290e8d2599e1b86f6a77fd1131bc678788c99a8a311"
Jul 16 22:27:30.552393 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:30.552324    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-92snh_odh-ai-gateway-infra(10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc"
Jul 16 22:27:31.559026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:31.558997    2555 scope.go:117] "RemoveContainer" containerID="6919fccd468b878585b16290e8d2599e1b86f6a77fd1131bc678788c99a8a311"
Jul 16 22:27:31.559442 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:31.559178    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-92snh_odh-ai-gateway-infra(10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc"
Jul 16 22:27:32.490022 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:32.489983    2555 scope.go:117] "RemoveContainer" containerID="4b222c2de47cbf3466e0c4276fcc7e8d1f1c3447a555fba5351ff4d2f6f4d864"
Jul 16 22:27:33.567682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:33.567646    2555 generic.go:356] "Generic (PLEG): container finished" podID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerID="4a00d91d8ab9a6196b4eedd18e2d2f21e1fc489cec374a7a778fdabb66f69824" exitCode=6
Jul 16 22:27:33.568111 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:33.567713    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" event={"ID":"83e6bc68-d3f1-4fe6-83fd-e4148a670808","Type":"ContainerDied","Data":"4a00d91d8ab9a6196b4eedd18e2d2f21e1fc489cec374a7a778fdabb66f69824"}
Jul 16 22:27:33.568111 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:33.567796    2555 scope.go:117] "RemoveContainer" containerID="4b222c2de47cbf3466e0c4276fcc7e8d1f1c3447a555fba5351ff4d2f6f4d864"
Jul 16 22:27:33.568111 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:33.568109    2555 scope.go:117] "RemoveContainer" containerID="4a00d91d8ab9a6196b4eedd18e2d2f21e1fc489cec374a7a778fdabb66f69824"
Jul 16 22:27:33.568328 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:33.568308    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s_odh-ai-gateway-infra(83e6bc68-d3f1-4fe6-83fd-e4148a670808)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808"
Jul 16 22:27:34.595145 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:34.595111    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"]
Jul 16 22:27:34.728231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:34.728202    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"
Jul 16 22:27:34.776932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:34.776898    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6bc68-d3f1-4fe6-83fd-e4148a670808-maas-api-service-ca\") pod \"83e6bc68-d3f1-4fe6-83fd-e4148a670808\" (UID: \"83e6bc68-d3f1-4fe6-83fd-e4148a670808\") "
Jul 16 22:27:34.777300 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:34.777276    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e6bc68-d3f1-4fe6-83fd-e4148a670808-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83e6bc68-d3f1-4fe6-83fd-e4148a670808" (UID: "83e6bc68-d3f1-4fe6-83fd-e4148a670808"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:34.877991 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:34.877875    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e6bc68-d3f1-4fe6-83fd-e4148a670808-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:27:35.327526 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327496    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"]
Jul 16 22:27:35.327794 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327782    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.327842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327796    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.327842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327804    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.327842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327810    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.327842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327824    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.327842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327829    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.328043 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327879    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.328043 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.327887    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:27:35.330807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.330790    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"
Jul 16 22:27:35.338636 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.338598    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"]
Jul 16 22:27:35.382765 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.382731    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f5a40-8fae-4773-8c27-db83e2f940e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl\" (UID: \"f65f5a40-8fae-4773-8c27-db83e2f940e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"
Jul 16 22:27:35.484188 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.484152    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f5a40-8fae-4773-8c27-db83e2f940e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl\" (UID: \"f65f5a40-8fae-4773-8c27-db83e2f940e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"
Jul 16 22:27:35.484776 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.484754    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f5a40-8fae-4773-8c27-db83e2f940e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl\" (UID: \"f65f5a40-8fae-4773-8c27-db83e2f940e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"
Jul 16 22:27:35.578746 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.578669    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"
Jul 16 22:27:35.578900 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.578671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s" event={"ID":"83e6bc68-d3f1-4fe6-83fd-e4148a670808","Type":"ContainerDied","Data":"95bfe3f47652f7a33c09e78eca2c5db254cef9ad44797e0b8e45b1c011704647"}
Jul 16 22:27:35.578900 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.578795    2555 scope.go:117] "RemoveContainer" containerID="4a00d91d8ab9a6196b4eedd18e2d2f21e1fc489cec374a7a778fdabb66f69824"
Jul 16 22:27:35.603308 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.603286    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"]
Jul 16 22:27:35.615313 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.615288    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-dpt7s"]
Jul 16 22:27:35.642074 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.642055    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"
Jul 16 22:27:35.981807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:35.981782    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"]
Jul 16 22:27:35.983142 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:27:35.983101    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65f5a40_8fae_4773_8c27_db83e2f940e4.slice/crio-2265b64d6d3382c4a83ca73f4934dcf188cb1722e2465a7911a3c11ce5e7be56 WatchSource:0}: Error finding container 2265b64d6d3382c4a83ca73f4934dcf188cb1722e2465a7911a3c11ce5e7be56: Status 404 returned error can't find the container with id 2265b64d6d3382c4a83ca73f4934dcf188cb1722e2465a7911a3c11ce5e7be56
Jul 16 22:27:36.495547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:36.495463    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" path="/var/lib/kubelet/pods/83e6bc68-d3f1-4fe6-83fd-e4148a670808/volumes"
Jul 16 22:27:36.583825 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:36.583786    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" event={"ID":"f65f5a40-8fae-4773-8c27-db83e2f940e4","Type":"ContainerStarted","Data":"ca1908188e4a4ced33b1455394847900056064671f8e4ab9d92d7e9d9899da45"}
Jul 16 22:27:36.583998 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:36.583844    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" event={"ID":"f65f5a40-8fae-4773-8c27-db83e2f940e4","Type":"ContainerStarted","Data":"2265b64d6d3382c4a83ca73f4934dcf188cb1722e2465a7911a3c11ce5e7be56"}
Jul 16 22:27:36.603805 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:36.603756    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" podStartSLOduration=1.6037414490000002 podStartE2EDuration="1.603741449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:36.601991352 +0000 UTC m=+2412.595155505" watchObservedRunningTime="2026-07-16 22:27:36.603741449 +0000 UTC m=+2412.596905600"
Jul 16 22:27:37.590902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:37.590862    2555 generic.go:356] "Generic (PLEG): container finished" podID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerID="ca1908188e4a4ced33b1455394847900056064671f8e4ab9d92d7e9d9899da45" exitCode=6
Jul 16 22:27:37.591131 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:37.590944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" event={"ID":"f65f5a40-8fae-4773-8c27-db83e2f940e4","Type":"ContainerDied","Data":"ca1908188e4a4ced33b1455394847900056064671f8e4ab9d92d7e9d9899da45"}
Jul 16 22:27:37.591209 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:37.591190    2555 scope.go:117] "RemoveContainer" containerID="ca1908188e4a4ced33b1455394847900056064671f8e4ab9d92d7e9d9899da45"
Jul 16 22:27:38.597042 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:38.597013    2555 generic.go:356] "Generic (PLEG): container finished" podID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerID="3d562bae9f7d40f386f33733eb8390b6cf0a97fe5281ecc2894cc7b279bd63b1" exitCode=6
Jul 16 22:27:38.597485 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:38.597098    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" event={"ID":"f65f5a40-8fae-4773-8c27-db83e2f940e4","Type":"ContainerDied","Data":"3d562bae9f7d40f386f33733eb8390b6cf0a97fe5281ecc2894cc7b279bd63b1"}
Jul 16 22:27:38.597485 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:38.597160    2555 scope.go:117] "RemoveContainer" containerID="ca1908188e4a4ced33b1455394847900056064671f8e4ab9d92d7e9d9899da45"
Jul 16 22:27:38.597485 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:38.597251    2555 scope.go:117] "RemoveContainer" containerID="3d562bae9f7d40f386f33733eb8390b6cf0a97fe5281ecc2894cc7b279bd63b1"
Jul 16 22:27:38.597485 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:38.597465    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl_odh-ai-gateway-infra(f65f5a40-8fae-4773-8c27-db83e2f940e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4"
Jul 16 22:27:39.602567 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:39.602534    2555 scope.go:117] "RemoveContainer" containerID="3d562bae9f7d40f386f33733eb8390b6cf0a97fe5281ecc2894cc7b279bd63b1"
Jul 16 22:27:39.602932 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:39.602738    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl_odh-ai-gateway-infra(f65f5a40-8fae-4773-8c27-db83e2f940e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4"
Jul 16 22:27:44.492240 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:44.492193    2555 scope.go:117] "RemoveContainer" containerID="6919fccd468b878585b16290e8d2599e1b86f6a77fd1131bc678788c99a8a311"
Jul 16 22:27:45.629192 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:45.629109    2555 generic.go:356] "Generic (PLEG): container finished" podID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerID="1a4c5012ae25a0656e268778c8a7ebd5842e48202db5585c03b60afa848f8331" exitCode=7
Jul 16 22:27:45.629569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:45.629188    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" event={"ID":"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc","Type":"ContainerDied","Data":"1a4c5012ae25a0656e268778c8a7ebd5842e48202db5585c03b60afa848f8331"}
Jul 16 22:27:45.629569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:45.629250    2555 scope.go:117] "RemoveContainer" containerID="6919fccd468b878585b16290e8d2599e1b86f6a77fd1131bc678788c99a8a311"
Jul 16 22:27:45.629569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:45.629498    2555 scope.go:117] "RemoveContainer" containerID="1a4c5012ae25a0656e268778c8a7ebd5842e48202db5585c03b60afa848f8331"
Jul 16 22:27:45.629716 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:45.629690    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-92snh_odh-ai-gateway-infra(10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc"
Jul 16 22:27:46.659022 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:46.658989    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"]
Jul 16 22:27:46.789555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:46.789528    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"
Jul 16 22:27:46.883252 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:46.883219    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc-maas-api-service-ca\") pod \"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc\" (UID: \"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc\") "
Jul 16 22:27:46.883556 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:46.883530    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" (UID: "10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:46.984786 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:46.984697    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:27:47.640086 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:47.640051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh" event={"ID":"10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc","Type":"ContainerDied","Data":"021e58d6c21abc4406b584218c364667ab70d339e8557a7fbea6c15553ca155d"}
Jul 16 22:27:47.640086 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:47.640074    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"
Jul 16 22:27:47.640336 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:47.640112    2555 scope.go:117] "RemoveContainer" containerID="1a4c5012ae25a0656e268778c8a7ebd5842e48202db5585c03b60afa848f8331"
Jul 16 22:27:47.665063 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:47.665033    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"]
Jul 16 22:27:47.669944 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:47.669920    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-92snh"]
Jul 16 22:27:48.495213 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:48.495181    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" path="/var/lib/kubelet/pods/10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc/volumes"
Jul 16 22:27:52.490152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:52.490115    2555 scope.go:117] "RemoveContainer" containerID="3d562bae9f7d40f386f33733eb8390b6cf0a97fe5281ecc2894cc7b279bd63b1"
Jul 16 22:27:53.667885 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:53.667847    2555 generic.go:356] "Generic (PLEG): container finished" podID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerID="8504508f7ad8aeef8695428141502266dd7a1327b419472d8fa7df05b0666fb4" exitCode=6
Jul 16 22:27:53.668344 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:53.667915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" event={"ID":"f65f5a40-8fae-4773-8c27-db83e2f940e4","Type":"ContainerDied","Data":"8504508f7ad8aeef8695428141502266dd7a1327b419472d8fa7df05b0666fb4"}
Jul 16 22:27:53.668344 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:53.667992    2555 scope.go:117] "RemoveContainer" containerID="3d562bae9f7d40f386f33733eb8390b6cf0a97fe5281ecc2894cc7b279bd63b1"
Jul 16 22:27:53.668344 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:53.668274    2555 scope.go:117] "RemoveContainer" containerID="8504508f7ad8aeef8695428141502266dd7a1327b419472d8fa7df05b0666fb4"
Jul 16 22:27:53.668503 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:27:53.668481    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl_odh-ai-gateway-infra(f65f5a40-8fae-4773-8c27-db83e2f940e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4"
Jul 16 22:27:54.693815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:54.693782    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"]
Jul 16 22:27:54.822153 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:54.822129    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"
Jul 16 22:27:54.956691 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:54.956597    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f5a40-8fae-4773-8c27-db83e2f940e4-maas-api-service-ca\") pod \"f65f5a40-8fae-4773-8c27-db83e2f940e4\" (UID: \"f65f5a40-8fae-4773-8c27-db83e2f940e4\") "
Jul 16 22:27:54.956973 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:54.956934    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65f5a40-8fae-4773-8c27-db83e2f940e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f65f5a40-8fae-4773-8c27-db83e2f940e4" (UID: "f65f5a40-8fae-4773-8c27-db83e2f940e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:55.058079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:55.058048    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f5a40-8fae-4773-8c27-db83e2f940e4-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:27:55.679079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:55.679053    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"
Jul 16 22:27:55.679079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:55.679075    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl" event={"ID":"f65f5a40-8fae-4773-8c27-db83e2f940e4","Type":"ContainerDied","Data":"2265b64d6d3382c4a83ca73f4934dcf188cb1722e2465a7911a3c11ce5e7be56"}
Jul 16 22:27:55.679321 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:55.679130    2555 scope.go:117] "RemoveContainer" containerID="8504508f7ad8aeef8695428141502266dd7a1327b419472d8fa7df05b0666fb4"
Jul 16 22:27:55.699336 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:55.699310    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"]
Jul 16 22:27:55.707376 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:55.707352    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-mk2bl"]
Jul 16 22:27:56.494858 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:27:56.494828    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" path="/var/lib/kubelet/pods/f65f5a40-8fae-4773-8c27-db83e2f940e4/volumes"
Jul 16 22:28:15.028850 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.028820    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"]
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029122    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029134    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029141    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029147    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029158    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029164    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029183    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029188    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029235    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029241    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029247    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029254    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029260    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:15.029287 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.029265    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e6bc68-d3f1-4fe6-83fd-e4148a670808" containerName="revoke-keys"
Jul 16 22:28:15.032228 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.032213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"
Jul 16 22:28:15.034900 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.034880    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:28:15.034900 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.034887    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:28:15.040165 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.040139    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"]
Jul 16 22:28:15.129196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.129154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f733103-0af2-4fe7-ac6f-2b193e5759d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7\" (UID: \"4f733103-0af2-4fe7-ac6f-2b193e5759d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"
Jul 16 22:28:15.230244 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.230205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f733103-0af2-4fe7-ac6f-2b193e5759d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7\" (UID: \"4f733103-0af2-4fe7-ac6f-2b193e5759d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"
Jul 16 22:28:15.230822 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.230800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f733103-0af2-4fe7-ac6f-2b193e5759d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7\" (UID: \"4f733103-0af2-4fe7-ac6f-2b193e5759d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"
Jul 16 22:28:15.343282 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.343252    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"
Jul 16 22:28:15.468792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.468763    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"]
Jul 16 22:28:15.471341 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:28:15.471306    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f733103_0af2_4fe7_ac6f_2b193e5759d3.slice/crio-30d9f5364a7915b28c239ba545863ff2729ff1ae3101610f1d145495f8692a00 WatchSource:0}: Error finding container 30d9f5364a7915b28c239ba545863ff2729ff1ae3101610f1d145495f8692a00: Status 404 returned error can't find the container with id 30d9f5364a7915b28c239ba545863ff2729ff1ae3101610f1d145495f8692a00
Jul 16 22:28:15.763387 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.763351    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" event={"ID":"4f733103-0af2-4fe7-ac6f-2b193e5759d3","Type":"ContainerStarted","Data":"9fd80a9bfc7e6afc774a893d3d357faafd96949e44ca1ee3c3a02e907f204b43"}
Jul 16 22:28:15.763542 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.763408    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" event={"ID":"4f733103-0af2-4fe7-ac6f-2b193e5759d3","Type":"ContainerStarted","Data":"30d9f5364a7915b28c239ba545863ff2729ff1ae3101610f1d145495f8692a00"}
Jul 16 22:28:15.779919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:15.779874    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" podStartSLOduration=0.779858854 podStartE2EDuration="779.858854ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:15.779425468 +0000 UTC m=+2451.772589621" watchObservedRunningTime="2026-07-16 22:28:15.779858854 +0000 UTC m=+2451.773023006"
Jul 16 22:28:16.768755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:16.768725    2555 generic.go:356] "Generic (PLEG): container finished" podID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerID="9fd80a9bfc7e6afc774a893d3d357faafd96949e44ca1ee3c3a02e907f204b43" exitCode=6
Jul 16 22:28:16.769163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:16.768790    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" event={"ID":"4f733103-0af2-4fe7-ac6f-2b193e5759d3","Type":"ContainerDied","Data":"9fd80a9bfc7e6afc774a893d3d357faafd96949e44ca1ee3c3a02e907f204b43"}
Jul 16 22:28:16.769163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:16.769008    2555 scope.go:117] "RemoveContainer" containerID="9fd80a9bfc7e6afc774a893d3d357faafd96949e44ca1ee3c3a02e907f204b43"
Jul 16 22:28:17.774349 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:17.774263    2555 generic.go:356] "Generic (PLEG): container finished" podID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerID="a49457bff3193992d365b4680177c65e6487b618e7b5bf36b908a29aa2180f72" exitCode=6
Jul 16 22:28:17.774846 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:17.774347    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" event={"ID":"4f733103-0af2-4fe7-ac6f-2b193e5759d3","Type":"ContainerDied","Data":"a49457bff3193992d365b4680177c65e6487b618e7b5bf36b908a29aa2180f72"}
Jul 16 22:28:17.774846 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:17.774404    2555 scope.go:117] "RemoveContainer" containerID="9fd80a9bfc7e6afc774a893d3d357faafd96949e44ca1ee3c3a02e907f204b43"
Jul 16 22:28:17.774846 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:17.774583    2555 scope.go:117] "RemoveContainer" containerID="a49457bff3193992d365b4680177c65e6487b618e7b5bf36b908a29aa2180f72"
Jul 16 22:28:17.774846 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:17.774806    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7_odh-ai-gateway-infra(4f733103-0af2-4fe7-ac6f-2b193e5759d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3"
Jul 16 22:28:18.052908 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.052818    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"]
Jul 16 22:28:18.053561 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.053538    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:18.053674 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.053564    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:18.053674 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.053590    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:18.053674 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.053598    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65f5a40-8fae-4773-8c27-db83e2f940e4" containerName="revoke-keys"
Jul 16 22:28:18.053830 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.053772    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="10fcd66e-9fa5-4879-9ba4-96ed9ba1f2fc" containerName="revoke-keys"
Jul 16 22:28:18.058536 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.058512    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"
Jul 16 22:28:18.063694 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.063670    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"]
Jul 16 22:28:18.158864 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.158829    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03dd89f8-de62-4eb3-86fa-69d8e8dbe182-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw\" (UID: \"03dd89f8-de62-4eb3-86fa-69d8e8dbe182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"
Jul 16 22:28:18.259326 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.259289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03dd89f8-de62-4eb3-86fa-69d8e8dbe182-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw\" (UID: \"03dd89f8-de62-4eb3-86fa-69d8e8dbe182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"
Jul 16 22:28:18.259894 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.259875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03dd89f8-de62-4eb3-86fa-69d8e8dbe182-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw\" (UID: \"03dd89f8-de62-4eb3-86fa-69d8e8dbe182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"
Jul 16 22:28:18.370217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.370185    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"
Jul 16 22:28:18.497439 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.497251    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"]
Jul 16 22:28:18.500026 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:28:18.500000    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dd89f8_de62_4eb3_86fa_69d8e8dbe182.slice/crio-ac3d61286c5b99c016da94582b16ba7797b124324950bc874f4d47d4acfeb0d7 WatchSource:0}: Error finding container ac3d61286c5b99c016da94582b16ba7797b124324950bc874f4d47d4acfeb0d7: Status 404 returned error can't find the container with id ac3d61286c5b99c016da94582b16ba7797b124324950bc874f4d47d4acfeb0d7
Jul 16 22:28:18.780395 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.780361    2555 scope.go:117] "RemoveContainer" containerID="a49457bff3193992d365b4680177c65e6487b618e7b5bf36b908a29aa2180f72"
Jul 16 22:28:18.780842 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:18.780601    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7_odh-ai-gateway-infra(4f733103-0af2-4fe7-ac6f-2b193e5759d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3"
Jul 16 22:28:18.781739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.781704    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" event={"ID":"03dd89f8-de62-4eb3-86fa-69d8e8dbe182","Type":"ContainerStarted","Data":"3741aea5bd7dd19005e4ee67ee9a3a28475fb34e06035f0ba4364b4088048313"}
Jul 16 22:28:18.781836 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.781765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" event={"ID":"03dd89f8-de62-4eb3-86fa-69d8e8dbe182","Type":"ContainerStarted","Data":"ac3d61286c5b99c016da94582b16ba7797b124324950bc874f4d47d4acfeb0d7"}
Jul 16 22:28:18.811077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:18.811022    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" podStartSLOduration=0.811004261 podStartE2EDuration="811.004261ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:18.809476508 +0000 UTC m=+2454.802640660" watchObservedRunningTime="2026-07-16 22:28:18.811004261 +0000 UTC m=+2454.804168413"
Jul 16 22:28:19.787625 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:19.787594    2555 generic.go:356] "Generic (PLEG): container finished" podID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerID="3741aea5bd7dd19005e4ee67ee9a3a28475fb34e06035f0ba4364b4088048313" exitCode=7
Jul 16 22:28:19.788033 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:19.787653    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" event={"ID":"03dd89f8-de62-4eb3-86fa-69d8e8dbe182","Type":"ContainerDied","Data":"3741aea5bd7dd19005e4ee67ee9a3a28475fb34e06035f0ba4364b4088048313"}
Jul 16 22:28:19.788033 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:19.787839    2555 scope.go:117] "RemoveContainer" containerID="3741aea5bd7dd19005e4ee67ee9a3a28475fb34e06035f0ba4364b4088048313"
Jul 16 22:28:20.792895 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:20.792810    2555 generic.go:356] "Generic (PLEG): container finished" podID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerID="555357fea00495cd18e6c4a4aaf0a0ecc6ff25ad3baa922cf42bb49f8f8e6f13" exitCode=7
Jul 16 22:28:20.792895 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:20.792859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" event={"ID":"03dd89f8-de62-4eb3-86fa-69d8e8dbe182","Type":"ContainerDied","Data":"555357fea00495cd18e6c4a4aaf0a0ecc6ff25ad3baa922cf42bb49f8f8e6f13"}
Jul 16 22:28:20.793388 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:20.792907    2555 scope.go:117] "RemoveContainer" containerID="3741aea5bd7dd19005e4ee67ee9a3a28475fb34e06035f0ba4364b4088048313"
Jul 16 22:28:20.793388 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:20.793084    2555 scope.go:117] "RemoveContainer" containerID="555357fea00495cd18e6c4a4aaf0a0ecc6ff25ad3baa922cf42bb49f8f8e6f13"
Jul 16 22:28:20.793388 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:20.793295    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw_odh-ai-gateway-infra(03dd89f8-de62-4eb3-86fa-69d8e8dbe182)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182"
Jul 16 22:28:21.798643 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:21.798617    2555 scope.go:117] "RemoveContainer" containerID="555357fea00495cd18e6c4a4aaf0a0ecc6ff25ad3baa922cf42bb49f8f8e6f13"
Jul 16 22:28:21.799065 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:21.798800    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw_odh-ai-gateway-infra(03dd89f8-de62-4eb3-86fa-69d8e8dbe182)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182"
Jul 16 22:28:26.753934 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:26.753893    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"]
Jul 16 22:28:26.758757 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:26.758729    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"
Jul 16 22:28:26.764177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:26.764127    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"]
Jul 16 22:28:26.836351 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:26.836316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68ac7cc-44c1-4713-b99f-5c85aafcb554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75\" (UID: \"e68ac7cc-44c1-4713-b99f-5c85aafcb554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"
Jul 16 22:28:26.937153 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:26.937121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68ac7cc-44c1-4713-b99f-5c85aafcb554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75\" (UID: \"e68ac7cc-44c1-4713-b99f-5c85aafcb554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"
Jul 16 22:28:26.937761 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:26.937741    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68ac7cc-44c1-4713-b99f-5c85aafcb554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75\" (UID: \"e68ac7cc-44c1-4713-b99f-5c85aafcb554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"
Jul 16 22:28:27.070374 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:27.070291    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"
Jul 16 22:28:27.196814 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:27.196789    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"]
Jul 16 22:28:27.198485 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:28:27.198455    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68ac7cc_44c1_4713_b99f_5c85aafcb554.slice/crio-294d7f09210119565c3a4bb0a698b655fd07dd43021695aa943943dfb57a5744 WatchSource:0}: Error finding container 294d7f09210119565c3a4bb0a698b655fd07dd43021695aa943943dfb57a5744: Status 404 returned error can't find the container with id 294d7f09210119565c3a4bb0a698b655fd07dd43021695aa943943dfb57a5744
Jul 16 22:28:27.824368 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:27.824284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" event={"ID":"e68ac7cc-44c1-4713-b99f-5c85aafcb554","Type":"ContainerStarted","Data":"2099b5d9a1e5342a0a613c8b2989492a660c33edff45b69ebf63b4e1f0c1dc54"}
Jul 16 22:28:27.824368 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:27.824339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" event={"ID":"e68ac7cc-44c1-4713-b99f-5c85aafcb554","Type":"ContainerStarted","Data":"294d7f09210119565c3a4bb0a698b655fd07dd43021695aa943943dfb57a5744"}
Jul 16 22:28:27.840344 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:27.840289    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" podStartSLOduration=1.840268115 podStartE2EDuration="1.840268115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:27.838405436 +0000 UTC m=+2463.831569589" watchObservedRunningTime="2026-07-16 22:28:27.840268115 +0000 UTC m=+2463.833432269"
Jul 16 22:28:28.829327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:28.829287    2555 generic.go:356] "Generic (PLEG): container finished" podID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerID="2099b5d9a1e5342a0a613c8b2989492a660c33edff45b69ebf63b4e1f0c1dc54" exitCode=6
Jul 16 22:28:28.829728 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:28.829372    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" event={"ID":"e68ac7cc-44c1-4713-b99f-5c85aafcb554","Type":"ContainerDied","Data":"2099b5d9a1e5342a0a613c8b2989492a660c33edff45b69ebf63b4e1f0c1dc54"}
Jul 16 22:28:28.829728 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:28.829564    2555 scope.go:117] "RemoveContainer" containerID="2099b5d9a1e5342a0a613c8b2989492a660c33edff45b69ebf63b4e1f0c1dc54"
Jul 16 22:28:29.835562 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:29.835530    2555 generic.go:356] "Generic (PLEG): container finished" podID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerID="b091a82760d60a44b6cc04abd7d881923ac3a4bbfcff7bbc2126ec7662ed038d" exitCode=6
Jul 16 22:28:29.835999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:29.835573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" event={"ID":"e68ac7cc-44c1-4713-b99f-5c85aafcb554","Type":"ContainerDied","Data":"b091a82760d60a44b6cc04abd7d881923ac3a4bbfcff7bbc2126ec7662ed038d"}
Jul 16 22:28:29.835999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:29.835628    2555 scope.go:117] "RemoveContainer" containerID="2099b5d9a1e5342a0a613c8b2989492a660c33edff45b69ebf63b4e1f0c1dc54"
Jul 16 22:28:29.835999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:29.835819    2555 scope.go:117] "RemoveContainer" containerID="b091a82760d60a44b6cc04abd7d881923ac3a4bbfcff7bbc2126ec7662ed038d"
Jul 16 22:28:29.836148 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:29.836030    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75_odh-ai-gateway-infra(e68ac7cc-44c1-4713-b99f-5c85aafcb554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554"
Jul 16 22:28:30.841316 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:30.841287    2555 scope.go:117] "RemoveContainer" containerID="b091a82760d60a44b6cc04abd7d881923ac3a4bbfcff7bbc2126ec7662ed038d"
Jul 16 22:28:30.841717 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:30.841462    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75_odh-ai-gateway-infra(e68ac7cc-44c1-4713-b99f-5c85aafcb554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554"
Jul 16 22:28:31.490252 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:31.490226    2555 scope.go:117] "RemoveContainer" containerID="a49457bff3193992d365b4680177c65e6487b618e7b5bf36b908a29aa2180f72"
Jul 16 22:28:31.846433 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:31.846398    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" event={"ID":"4f733103-0af2-4fe7-ac6f-2b193e5759d3","Type":"ContainerStarted","Data":"cace6c454cec78d2c788ff061b96968723b2bea4de9fafac6ccbb857a710635f"}
Jul 16 22:28:32.853505 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:32.853473    2555 generic.go:356] "Generic (PLEG): container finished" podID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerID="cace6c454cec78d2c788ff061b96968723b2bea4de9fafac6ccbb857a710635f" exitCode=6
Jul 16 22:28:32.853932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:32.853552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" event={"ID":"4f733103-0af2-4fe7-ac6f-2b193e5759d3","Type":"ContainerDied","Data":"cace6c454cec78d2c788ff061b96968723b2bea4de9fafac6ccbb857a710635f"}
Jul 16 22:28:32.853932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:32.853617    2555 scope.go:117] "RemoveContainer" containerID="a49457bff3193992d365b4680177c65e6487b618e7b5bf36b908a29aa2180f72"
Jul 16 22:28:32.853932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:32.853909    2555 scope.go:117] "RemoveContainer" containerID="cace6c454cec78d2c788ff061b96968723b2bea4de9fafac6ccbb857a710635f"
Jul 16 22:28:32.854116 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:32.854096    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7_odh-ai-gateway-infra(4f733103-0af2-4fe7-ac6f-2b193e5759d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3"
Jul 16 22:28:32.882346 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:32.882321    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"]
Jul 16 22:28:33.991938 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:33.991913    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"
Jul 16 22:28:34.106084 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.106051    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f733103-0af2-4fe7-ac6f-2b193e5759d3-maas-api-service-ca\") pod \"4f733103-0af2-4fe7-ac6f-2b193e5759d3\" (UID: \"4f733103-0af2-4fe7-ac6f-2b193e5759d3\") "
Jul 16 22:28:34.106467 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.106441    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f733103-0af2-4fe7-ac6f-2b193e5759d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f733103-0af2-4fe7-ac6f-2b193e5759d3" (UID: "4f733103-0af2-4fe7-ac6f-2b193e5759d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:34.207277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.207182    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f733103-0af2-4fe7-ac6f-2b193e5759d3-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:28:34.491654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.491570    2555 scope.go:117] "RemoveContainer" containerID="555357fea00495cd18e6c4a4aaf0a0ecc6ff25ad3baa922cf42bb49f8f8e6f13"
Jul 16 22:28:34.865600 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.865569    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"
Jul 16 22:28:34.865600 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.865590    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7" event={"ID":"4f733103-0af2-4fe7-ac6f-2b193e5759d3","Type":"ContainerDied","Data":"30d9f5364a7915b28c239ba545863ff2729ff1ae3101610f1d145495f8692a00"}
Jul 16 22:28:34.865866 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.865655    2555 scope.go:117] "RemoveContainer" containerID="cace6c454cec78d2c788ff061b96968723b2bea4de9fafac6ccbb857a710635f"
Jul 16 22:28:34.867606 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.867578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" event={"ID":"03dd89f8-de62-4eb3-86fa-69d8e8dbe182","Type":"ContainerStarted","Data":"11b95c72be1ab6b78587663eaee74d98ccb4d8e336b7692a24296b93f4cc2171"}
Jul 16 22:28:34.882894 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.882869    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"]
Jul 16 22:28:34.888585 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:34.888562    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-fqnp7"]
Jul 16 22:28:35.874236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:35.874205    2555 generic.go:356] "Generic (PLEG): container finished" podID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerID="11b95c72be1ab6b78587663eaee74d98ccb4d8e336b7692a24296b93f4cc2171" exitCode=7
Jul 16 22:28:35.874696 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:35.874256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" event={"ID":"03dd89f8-de62-4eb3-86fa-69d8e8dbe182","Type":"ContainerDied","Data":"11b95c72be1ab6b78587663eaee74d98ccb4d8e336b7692a24296b93f4cc2171"}
Jul 16 22:28:35.874696 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:35.874314    2555 scope.go:117] "RemoveContainer" containerID="555357fea00495cd18e6c4a4aaf0a0ecc6ff25ad3baa922cf42bb49f8f8e6f13"
Jul 16 22:28:35.874696 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:35.874586    2555 scope.go:117] "RemoveContainer" containerID="11b95c72be1ab6b78587663eaee74d98ccb4d8e336b7692a24296b93f4cc2171"
Jul 16 22:28:35.874922 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:35.874788    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw_odh-ai-gateway-infra(03dd89f8-de62-4eb3-86fa-69d8e8dbe182)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182"
Jul 16 22:28:35.912694 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:35.912665    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"]
Jul 16 22:28:36.495387 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:36.495358    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" path="/var/lib/kubelet/pods/4f733103-0af2-4fe7-ac6f-2b193e5759d3/volumes"
Jul 16 22:28:37.011418 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.011395    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"
Jul 16 22:28:37.134758 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.134677    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03dd89f8-de62-4eb3-86fa-69d8e8dbe182-maas-api-service-ca\") pod \"03dd89f8-de62-4eb3-86fa-69d8e8dbe182\" (UID: \"03dd89f8-de62-4eb3-86fa-69d8e8dbe182\") "
Jul 16 22:28:37.135078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.135053    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03dd89f8-de62-4eb3-86fa-69d8e8dbe182-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03dd89f8-de62-4eb3-86fa-69d8e8dbe182" (UID: "03dd89f8-de62-4eb3-86fa-69d8e8dbe182"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:37.235750 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.235704    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03dd89f8-de62-4eb3-86fa-69d8e8dbe182-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:28:37.884203 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.884170    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"
Jul 16 22:28:37.884355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.884167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw" event={"ID":"03dd89f8-de62-4eb3-86fa-69d8e8dbe182","Type":"ContainerDied","Data":"ac3d61286c5b99c016da94582b16ba7797b124324950bc874f4d47d4acfeb0d7"}
Jul 16 22:28:37.884355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.884319    2555 scope.go:117] "RemoveContainer" containerID="11b95c72be1ab6b78587663eaee74d98ccb4d8e336b7692a24296b93f4cc2171"
Jul 16 22:28:37.905538 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.905513    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"]
Jul 16 22:28:37.906811 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:37.906791    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bj5jw"]
Jul 16 22:28:38.494878 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:38.494845    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" path="/var/lib/kubelet/pods/03dd89f8-de62-4eb3-86fa-69d8e8dbe182/volumes"
Jul 16 22:28:44.491327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:44.491295    2555 scope.go:117] "RemoveContainer" containerID="b091a82760d60a44b6cc04abd7d881923ac3a4bbfcff7bbc2126ec7662ed038d"
Jul 16 22:28:44.913455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:44.913421    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" event={"ID":"e68ac7cc-44c1-4713-b99f-5c85aafcb554","Type":"ContainerStarted","Data":"113b6e8e6dcabbfec6ed156bb59585cd7a5df2f02e0ee18364e5a8daf4856961"}
Jul 16 22:28:45.919869 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:45.919838    2555 generic.go:356] "Generic (PLEG): container finished" podID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerID="113b6e8e6dcabbfec6ed156bb59585cd7a5df2f02e0ee18364e5a8daf4856961" exitCode=6
Jul 16 22:28:45.920302 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:45.919892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" event={"ID":"e68ac7cc-44c1-4713-b99f-5c85aafcb554","Type":"ContainerDied","Data":"113b6e8e6dcabbfec6ed156bb59585cd7a5df2f02e0ee18364e5a8daf4856961"}
Jul 16 22:28:45.920302 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:45.919939    2555 scope.go:117] "RemoveContainer" containerID="b091a82760d60a44b6cc04abd7d881923ac3a4bbfcff7bbc2126ec7662ed038d"
Jul 16 22:28:45.920302 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:45.920197    2555 scope.go:117] "RemoveContainer" containerID="113b6e8e6dcabbfec6ed156bb59585cd7a5df2f02e0ee18364e5a8daf4856961"
Jul 16 22:28:45.920407 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:28:45.920381    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75_odh-ai-gateway-infra(e68ac7cc-44c1-4713-b99f-5c85aafcb554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554"
Jul 16 22:28:45.946236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:45.946206    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"]
Jul 16 22:28:47.052302 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.052278    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"
Jul 16 22:28:47.118998 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.118947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68ac7cc-44c1-4713-b99f-5c85aafcb554-maas-api-service-ca\") pod \"e68ac7cc-44c1-4713-b99f-5c85aafcb554\" (UID: \"e68ac7cc-44c1-4713-b99f-5c85aafcb554\") "
Jul 16 22:28:47.119328 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.119306    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68ac7cc-44c1-4713-b99f-5c85aafcb554-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e68ac7cc-44c1-4713-b99f-5c85aafcb554" (UID: "e68ac7cc-44c1-4713-b99f-5c85aafcb554"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:47.220630 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.220532    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68ac7cc-44c1-4713-b99f-5c85aafcb554-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:28:47.930742 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.930708    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75" event={"ID":"e68ac7cc-44c1-4713-b99f-5c85aafcb554","Type":"ContainerDied","Data":"294d7f09210119565c3a4bb0a698b655fd07dd43021695aa943943dfb57a5744"}
Jul 16 22:28:47.930742 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.930737    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"
Jul 16 22:28:47.931039 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.930771    2555 scope.go:117] "RemoveContainer" containerID="113b6e8e6dcabbfec6ed156bb59585cd7a5df2f02e0ee18364e5a8daf4856961"
Jul 16 22:28:47.958570 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.958541    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"]
Jul 16 22:28:47.965383 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:47.965353    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pdf75"]
Jul 16 22:28:48.494633 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:28:48.494591    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" path="/var/lib/kubelet/pods/e68ac7cc-44c1-4713-b99f-5c85aafcb554/volumes"
Jul 16 22:29:26.816521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816484    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"]
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816808    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816819    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816832    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816837    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816843    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816849    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816854    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816859    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816865    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816870    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816884    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816889    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816895    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816900    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:29:26.816937 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816945    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:29:26.817451 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816969    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.817451 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816975    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:29:26.817451 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816982    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:29:26.817451 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816987    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:29:26.817451 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.816992    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:29:26.821315 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.821297    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"
Jul 16 22:29:26.824323 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.824299    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:29:26.824714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.824695    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:29:26.830991 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.830947    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"]
Jul 16 22:29:26.972560 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:26.972522    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a13af-dccd-4c9b-9379-feaa25fe7ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54\" (UID: \"322a13af-dccd-4c9b-9379-feaa25fe7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"
Jul 16 22:29:27.073033 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:27.072903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a13af-dccd-4c9b-9379-feaa25fe7ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54\" (UID: \"322a13af-dccd-4c9b-9379-feaa25fe7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"
Jul 16 22:29:27.073552 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:27.073532    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a13af-dccd-4c9b-9379-feaa25fe7ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54\" (UID: \"322a13af-dccd-4c9b-9379-feaa25fe7ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"
Jul 16 22:29:27.132723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:27.132685    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"
Jul 16 22:29:27.270394 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:27.270369    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"]
Jul 16 22:29:27.272731 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:29:27.272706    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322a13af_dccd_4c9b_9379_feaa25fe7ae3.slice/crio-ae485489c9611fd3540f38ceadb63932c094a1e153d4a95808780c6bf0e14c3a WatchSource:0}: Error finding container ae485489c9611fd3540f38ceadb63932c094a1e153d4a95808780c6bf0e14c3a: Status 404 returned error can't find the container with id ae485489c9611fd3540f38ceadb63932c094a1e153d4a95808780c6bf0e14c3a
Jul 16 22:29:28.093010 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:28.092945    2555 generic.go:356] "Generic (PLEG): container finished" podID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerID="39d758795890a28d874738ddf80f768e5c614bbf6e8b04f40fff2697c4d375d5" exitCode=6
Jul 16 22:29:28.093010 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:28.092993    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" event={"ID":"322a13af-dccd-4c9b-9379-feaa25fe7ae3","Type":"ContainerDied","Data":"39d758795890a28d874738ddf80f768e5c614bbf6e8b04f40fff2697c4d375d5"}
Jul 16 22:29:28.093443 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:28.093046    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" event={"ID":"322a13af-dccd-4c9b-9379-feaa25fe7ae3","Type":"ContainerStarted","Data":"ae485489c9611fd3540f38ceadb63932c094a1e153d4a95808780c6bf0e14c3a"}
Jul 16 22:29:28.093443 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:28.093213    2555 scope.go:117] "RemoveContainer" containerID="39d758795890a28d874738ddf80f768e5c614bbf6e8b04f40fff2697c4d375d5"
Jul 16 22:29:29.099271 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:29.099242    2555 generic.go:356] "Generic (PLEG): container finished" podID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerID="56dfbf6cfd71e91f89523fd7f577b9f29a2bc9e9e459638b17da231cdf2644b7" exitCode=6
Jul 16 22:29:29.099734 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:29.099335    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" event={"ID":"322a13af-dccd-4c9b-9379-feaa25fe7ae3","Type":"ContainerDied","Data":"56dfbf6cfd71e91f89523fd7f577b9f29a2bc9e9e459638b17da231cdf2644b7"}
Jul 16 22:29:29.099734 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:29.099410    2555 scope.go:117] "RemoveContainer" containerID="39d758795890a28d874738ddf80f768e5c614bbf6e8b04f40fff2697c4d375d5"
Jul 16 22:29:29.099734 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:29.099509    2555 scope.go:117] "RemoveContainer" containerID="56dfbf6cfd71e91f89523fd7f577b9f29a2bc9e9e459638b17da231cdf2644b7"
Jul 16 22:29:29.099873 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:29:29.099742    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54_odh-ai-gateway-infra(322a13af-dccd-4c9b-9379-feaa25fe7ae3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3"
Jul 16 22:29:30.104719 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:30.104694    2555 scope.go:117] "RemoveContainer" containerID="56dfbf6cfd71e91f89523fd7f577b9f29a2bc9e9e459638b17da231cdf2644b7"
Jul 16 22:29:30.105131 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:29:30.104903    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54_odh-ai-gateway-infra(322a13af-dccd-4c9b-9379-feaa25fe7ae3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3"
Jul 16 22:29:42.490039 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:42.489904    2555 scope.go:117] "RemoveContainer" containerID="56dfbf6cfd71e91f89523fd7f577b9f29a2bc9e9e459638b17da231cdf2644b7"
Jul 16 22:29:43.158786 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:43.158762    2555 generic.go:356] "Generic (PLEG): container finished" podID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerID="0a937ad5b41e3380ea50ef227c51e26cfa8842303bc35f682fff4828be3baf9f" exitCode=6
Jul 16 22:29:43.158916 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:43.158827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" event={"ID":"322a13af-dccd-4c9b-9379-feaa25fe7ae3","Type":"ContainerDied","Data":"0a937ad5b41e3380ea50ef227c51e26cfa8842303bc35f682fff4828be3baf9f"}
Jul 16 22:29:43.158916 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:43.158889    2555 scope.go:117] "RemoveContainer" containerID="56dfbf6cfd71e91f89523fd7f577b9f29a2bc9e9e459638b17da231cdf2644b7"
Jul 16 22:29:43.159174 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:43.159160    2555 scope.go:117] "RemoveContainer" containerID="0a937ad5b41e3380ea50ef227c51e26cfa8842303bc35f682fff4828be3baf9f"
Jul 16 22:29:43.159420 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:29:43.159397    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54_odh-ai-gateway-infra(322a13af-dccd-4c9b-9379-feaa25fe7ae3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3"
Jul 16 22:29:44.200285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:44.200248    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"]
Jul 16 22:29:44.331353 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:44.331323    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"
Jul 16 22:29:44.397440 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:44.397407    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a13af-dccd-4c9b-9379-feaa25fe7ae3-maas-api-service-ca\") pod \"322a13af-dccd-4c9b-9379-feaa25fe7ae3\" (UID: \"322a13af-dccd-4c9b-9379-feaa25fe7ae3\") "
Jul 16 22:29:44.397780 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:44.397751    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322a13af-dccd-4c9b-9379-feaa25fe7ae3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "322a13af-dccd-4c9b-9379-feaa25fe7ae3" (UID: "322a13af-dccd-4c9b-9379-feaa25fe7ae3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:44.498602 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:44.498518    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a13af-dccd-4c9b-9379-feaa25fe7ae3-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:29:45.169062 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:45.169032    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"
Jul 16 22:29:45.169062 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:45.169043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54" event={"ID":"322a13af-dccd-4c9b-9379-feaa25fe7ae3","Type":"ContainerDied","Data":"ae485489c9611fd3540f38ceadb63932c094a1e153d4a95808780c6bf0e14c3a"}
Jul 16 22:29:45.169329 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:45.169110    2555 scope.go:117] "RemoveContainer" containerID="0a937ad5b41e3380ea50ef227c51e26cfa8842303bc35f682fff4828be3baf9f"
Jul 16 22:29:45.193952 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:45.193921    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"]
Jul 16 22:29:45.202284 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:45.202251    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-hdv54"]
Jul 16 22:29:46.494902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:29:46.494870    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" path="/var/lib/kubelet/pods/322a13af-dccd-4c9b-9379-feaa25fe7ae3/volumes"
Jul 16 22:30:45.246157 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246114    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"]
Jul 16 22:30:45.246580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246539    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:30:45.246580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246552    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:30:45.246580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246562    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:30:45.246580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246568    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:30:45.246580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246577    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246583    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246597    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246602    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246610    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246617    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246665    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="03dd89f8-de62-4eb3-86fa-69d8e8dbe182" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246672    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246678    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68ac7cc-44c1-4713-b99f-5c85aafcb554" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246685    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:45.246772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.246691    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f733103-0af2-4fe7-ac6f-2b193e5759d3" containerName="revoke-keys"
Jul 16 22:30:45.249764 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.249742    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"
Jul 16 22:30:45.252490 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.252472    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:30:45.252607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.252572    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:30:45.264530 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.264506    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"]
Jul 16 22:30:45.317415 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.317371    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf5ffd3-93ad-402d-9dc9-938d87497712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nmswc\" (UID: \"3bf5ffd3-93ad-402d-9dc9-938d87497712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"
Jul 16 22:30:45.417896 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.417859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf5ffd3-93ad-402d-9dc9-938d87497712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nmswc\" (UID: \"3bf5ffd3-93ad-402d-9dc9-938d87497712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"
Jul 16 22:30:45.418621 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.418599    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf5ffd3-93ad-402d-9dc9-938d87497712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nmswc\" (UID: \"3bf5ffd3-93ad-402d-9dc9-938d87497712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"
Jul 16 22:30:45.559783 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.559683    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"
Jul 16 22:30:45.696609 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:45.696586    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"]
Jul 16 22:30:45.699017 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:30:45.698979    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf5ffd3_93ad_402d_9dc9_938d87497712.slice/crio-27735e469b2b1edf0cf5b714bd9fa789526b7cd13cc156acfbf5dfe270259163 WatchSource:0}: Error finding container 27735e469b2b1edf0cf5b714bd9fa789526b7cd13cc156acfbf5dfe270259163: Status 404 returned error can't find the container with id 27735e469b2b1edf0cf5b714bd9fa789526b7cd13cc156acfbf5dfe270259163
Jul 16 22:30:46.425388 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.425347    2555 generic.go:356] "Generic (PLEG): container finished" podID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerID="e512c4198c1205dd6f5a89cd8d26f7fd3f18d65733aadf1a4d0f07241ba5a76e" exitCode=6
Jul 16 22:30:46.425833 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.425430    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" event={"ID":"3bf5ffd3-93ad-402d-9dc9-938d87497712","Type":"ContainerDied","Data":"e512c4198c1205dd6f5a89cd8d26f7fd3f18d65733aadf1a4d0f07241ba5a76e"}
Jul 16 22:30:46.425833 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.425479    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" event={"ID":"3bf5ffd3-93ad-402d-9dc9-938d87497712","Type":"ContainerStarted","Data":"27735e469b2b1edf0cf5b714bd9fa789526b7cd13cc156acfbf5dfe270259163"}
Jul 16 22:30:46.425833 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.425610    2555 scope.go:117] "RemoveContainer" containerID="e512c4198c1205dd6f5a89cd8d26f7fd3f18d65733aadf1a4d0f07241ba5a76e"
Jul 16 22:30:46.773402 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.773370    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"]
Jul 16 22:30:46.773768 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.773754    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="322a13af-dccd-4c9b-9379-feaa25fe7ae3" containerName="revoke-keys"
Jul 16 22:30:46.776628 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.776612    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"
Jul 16 22:30:46.804649 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.804617    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"]
Jul 16 22:30:46.830389 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.830361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df82361-1574-45cf-bbf9-3805527c5d4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p\" (UID: \"0df82361-1574-45cf-bbf9-3805527c5d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"
Jul 16 22:30:46.932012 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.931885    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df82361-1574-45cf-bbf9-3805527c5d4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p\" (UID: \"0df82361-1574-45cf-bbf9-3805527c5d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"
Jul 16 22:30:46.933132 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:46.933103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df82361-1574-45cf-bbf9-3805527c5d4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p\" (UID: \"0df82361-1574-45cf-bbf9-3805527c5d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"
Jul 16 22:30:47.086226 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:47.086192    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"
Jul 16 22:30:47.267552 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:47.267477    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"]
Jul 16 22:30:47.270356 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:30:47.270325    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df82361_1574_45cf_bbf9_3805527c5d4b.slice/crio-9ca990348519c00ff8c45025c155ed6cf8b7f9472c071f527436c83049fb8d6d WatchSource:0}: Error finding container 9ca990348519c00ff8c45025c155ed6cf8b7f9472c071f527436c83049fb8d6d: Status 404 returned error can't find the container with id 9ca990348519c00ff8c45025c155ed6cf8b7f9472c071f527436c83049fb8d6d
Jul 16 22:30:47.431751 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:47.431712    2555 generic.go:356] "Generic (PLEG): container finished" podID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerID="d66b302ca95bbb2ed82b302cfa81590a6a8a3dca828d3f5c54390ee2e01d2f73" exitCode=6
Jul 16 22:30:47.432198 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:47.431815    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" event={"ID":"3bf5ffd3-93ad-402d-9dc9-938d87497712","Type":"ContainerDied","Data":"d66b302ca95bbb2ed82b302cfa81590a6a8a3dca828d3f5c54390ee2e01d2f73"}
Jul 16 22:30:47.432198 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:47.431873    2555 scope.go:117] "RemoveContainer" containerID="e512c4198c1205dd6f5a89cd8d26f7fd3f18d65733aadf1a4d0f07241ba5a76e"
Jul 16 22:30:47.432198 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:47.432110    2555 scope.go:117] "RemoveContainer" containerID="d66b302ca95bbb2ed82b302cfa81590a6a8a3dca828d3f5c54390ee2e01d2f73"
Jul 16 22:30:47.432355 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:30:47.432336    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nmswc_odh-ai-gateway-infra(3bf5ffd3-93ad-402d-9dc9-938d87497712)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712"
Jul 16 22:30:47.433210 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:47.433187    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p" event={"ID":"0df82361-1574-45cf-bbf9-3805527c5d4b","Type":"ContainerStarted","Data":"9ca990348519c00ff8c45025c155ed6cf8b7f9472c071f527436c83049fb8d6d"}
Jul 16 22:30:48.439024 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:48.438983    2555 generic.go:356] "Generic (PLEG): container finished" podID="0df82361-1574-45cf-bbf9-3805527c5d4b" containerID="a3d8f58657e04342b7a0da6834a939476bb77c1bf49f77c45d8bfd14925c7053" exitCode=0
Jul 16 22:30:48.439512 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:48.439051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p" event={"ID":"0df82361-1574-45cf-bbf9-3805527c5d4b","Type":"ContainerDied","Data":"a3d8f58657e04342b7a0da6834a939476bb77c1bf49f77c45d8bfd14925c7053"}
Jul 16 22:30:48.441025 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:48.441007    2555 scope.go:117] "RemoveContainer" containerID="d66b302ca95bbb2ed82b302cfa81590a6a8a3dca828d3f5c54390ee2e01d2f73"
Jul 16 22:30:48.441193 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:30:48.441176    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nmswc_odh-ai-gateway-infra(3bf5ffd3-93ad-402d-9dc9-938d87497712)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712"
Jul 16 22:30:49.567362 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:49.567339    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"
Jul 16 22:30:49.655909 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:49.655866    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df82361-1574-45cf-bbf9-3805527c5d4b-maas-api-service-ca\") pod \"0df82361-1574-45cf-bbf9-3805527c5d4b\" (UID: \"0df82361-1574-45cf-bbf9-3805527c5d4b\") "
Jul 16 22:30:49.656259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:49.656236    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df82361-1574-45cf-bbf9-3805527c5d4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0df82361-1574-45cf-bbf9-3805527c5d4b" (UID: "0df82361-1574-45cf-bbf9-3805527c5d4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:49.757613 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:49.757525    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df82361-1574-45cf-bbf9-3805527c5d4b-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:30:50.450804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:50.450776    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p"
Jul 16 22:30:50.450804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:50.450795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p" event={"ID":"0df82361-1574-45cf-bbf9-3805527c5d4b","Type":"ContainerDied","Data":"9ca990348519c00ff8c45025c155ed6cf8b7f9472c071f527436c83049fb8d6d"}
Jul 16 22:30:50.451046 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:50.450842    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ca990348519c00ff8c45025c155ed6cf8b7f9472c071f527436c83049fb8d6d"
Jul 16 22:30:59.490466 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:30:59.490429    2555 scope.go:117] "RemoveContainer" containerID="d66b302ca95bbb2ed82b302cfa81590a6a8a3dca828d3f5c54390ee2e01d2f73"
Jul 16 22:31:00.493756 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:00.493653    2555 generic.go:356] "Generic (PLEG): container finished" podID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerID="dbd7897bf99e97e1dbae86a9b39973ea30aa953066ffc876b1b1570718647793" exitCode=6
Jul 16 22:31:00.494951 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:00.494927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" event={"ID":"3bf5ffd3-93ad-402d-9dc9-938d87497712","Type":"ContainerDied","Data":"dbd7897bf99e97e1dbae86a9b39973ea30aa953066ffc876b1b1570718647793"}
Jul 16 22:31:00.495028 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:00.495015    2555 scope.go:117] "RemoveContainer" containerID="d66b302ca95bbb2ed82b302cfa81590a6a8a3dca828d3f5c54390ee2e01d2f73"
Jul 16 22:31:00.495321 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:00.495304    2555 scope.go:117] "RemoveContainer" containerID="dbd7897bf99e97e1dbae86a9b39973ea30aa953066ffc876b1b1570718647793"
Jul 16 22:31:00.495542 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:00.495506    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nmswc_odh-ai-gateway-infra(3bf5ffd3-93ad-402d-9dc9-938d87497712)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712"
Jul 16 22:31:01.522019 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:01.521985    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"]
Jul 16 22:31:01.655923 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:01.655893    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"
Jul 16 22:31:01.762646 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:01.762609    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf5ffd3-93ad-402d-9dc9-938d87497712-maas-api-service-ca\") pod \"3bf5ffd3-93ad-402d-9dc9-938d87497712\" (UID: \"3bf5ffd3-93ad-402d-9dc9-938d87497712\") "
Jul 16 22:31:01.763015 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:01.762989    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf5ffd3-93ad-402d-9dc9-938d87497712-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bf5ffd3-93ad-402d-9dc9-938d87497712" (UID: "3bf5ffd3-93ad-402d-9dc9-938d87497712"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:01.864170 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:01.864131    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf5ffd3-93ad-402d-9dc9-938d87497712-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:31:02.505000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:02.504971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc" event={"ID":"3bf5ffd3-93ad-402d-9dc9-938d87497712","Type":"ContainerDied","Data":"27735e469b2b1edf0cf5b714bd9fa789526b7cd13cc156acfbf5dfe270259163"}
Jul 16 22:31:02.505149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:02.505028    2555 scope.go:117] "RemoveContainer" containerID="dbd7897bf99e97e1dbae86a9b39973ea30aa953066ffc876b1b1570718647793"
Jul 16 22:31:02.505149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:02.504980    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"
Jul 16 22:31:02.524388 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:02.524357    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"]
Jul 16 22:31:02.528532 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:02.528509    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nmswc"]
Jul 16 22:31:04.495110 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:04.495079    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" path="/var/lib/kubelet/pods/3bf5ffd3-93ad-402d-9dc9-938d87497712/volumes"
Jul 16 22:31:08.241070 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241037    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"]
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241345    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241357    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241371    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df82361-1574-45cf-bbf9-3805527c5d4b" containerName="revoke-keys"
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241376    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df82361-1574-45cf-bbf9-3805527c5d4b" containerName="revoke-keys"
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241395    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241401    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241412    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241417    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241463    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241474    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df82361-1574-45cf-bbf9-3805527c5d4b" containerName="revoke-keys"
Jul 16 22:31:08.241747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241482    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.241747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.241490    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf5ffd3-93ad-402d-9dc9-938d87497712" containerName="revoke-keys"
Jul 16 22:31:08.245575 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.245555    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"
Jul 16 22:31:08.248485 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.248465    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:31:08.248652 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.248632    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:31:08.254925 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.254900    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"]
Jul 16 22:31:08.419912 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.419870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe860a-b540-4945-b4d9-8d020d648494-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh\" (UID: \"33fe860a-b540-4945-b4d9-8d020d648494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"
Jul 16 22:31:08.520605 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.520517    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe860a-b540-4945-b4d9-8d020d648494-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh\" (UID: \"33fe860a-b540-4945-b4d9-8d020d648494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"
Jul 16 22:31:08.521146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.521128    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe860a-b540-4945-b4d9-8d020d648494-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh\" (UID: \"33fe860a-b540-4945-b4d9-8d020d648494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"
Jul 16 22:31:08.556346 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.556310    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"
Jul 16 22:31:08.686590 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:08.686557    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"]
Jul 16 22:31:08.689651 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:31:08.689623    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fe860a_b540_4945_b4d9_8d020d648494.slice/crio-0aa137580a18e5d1d8de006e31a24daa97a358f96801b6ce022dbfbcc73ab4e6 WatchSource:0}: Error finding container 0aa137580a18e5d1d8de006e31a24daa97a358f96801b6ce022dbfbcc73ab4e6: Status 404 returned error can't find the container with id 0aa137580a18e5d1d8de006e31a24daa97a358f96801b6ce022dbfbcc73ab4e6
Jul 16 22:31:09.533847 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:09.533766    2555 generic.go:356] "Generic (PLEG): container finished" podID="33fe860a-b540-4945-b4d9-8d020d648494" containerID="457f902482004a7578bf87c7297181d261ada4f16f9bc675f52a4ab7908cf642" exitCode=6
Jul 16 22:31:09.533847 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:09.533825    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" event={"ID":"33fe860a-b540-4945-b4d9-8d020d648494","Type":"ContainerDied","Data":"457f902482004a7578bf87c7297181d261ada4f16f9bc675f52a4ab7908cf642"}
Jul 16 22:31:09.534277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:09.533864    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" event={"ID":"33fe860a-b540-4945-b4d9-8d020d648494","Type":"ContainerStarted","Data":"0aa137580a18e5d1d8de006e31a24daa97a358f96801b6ce022dbfbcc73ab4e6"}
Jul 16 22:31:09.534277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:09.534092    2555 scope.go:117] "RemoveContainer" containerID="457f902482004a7578bf87c7297181d261ada4f16f9bc675f52a4ab7908cf642"
Jul 16 22:31:10.539743 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:10.539654    2555 generic.go:356] "Generic (PLEG): container finished" podID="33fe860a-b540-4945-b4d9-8d020d648494" containerID="75d65ad774ada5d151590f644b58fe8f0f6f5d84cbc371b67e21ab54f707c00d" exitCode=6
Jul 16 22:31:10.539743 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:10.539721    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" event={"ID":"33fe860a-b540-4945-b4d9-8d020d648494","Type":"ContainerDied","Data":"75d65ad774ada5d151590f644b58fe8f0f6f5d84cbc371b67e21ab54f707c00d"}
Jul 16 22:31:10.540299 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:10.539769    2555 scope.go:117] "RemoveContainer" containerID="457f902482004a7578bf87c7297181d261ada4f16f9bc675f52a4ab7908cf642"
Jul 16 22:31:10.540299 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:10.539973    2555 scope.go:117] "RemoveContainer" containerID="75d65ad774ada5d151590f644b58fe8f0f6f5d84cbc371b67e21ab54f707c00d"
Jul 16 22:31:10.540299 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:10.540224    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh_odh-ai-gateway-infra(33fe860a-b540-4945-b4d9-8d020d648494)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" podUID="33fe860a-b540-4945-b4d9-8d020d648494"
Jul 16 22:31:11.546120 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:11.546092    2555 scope.go:117] "RemoveContainer" containerID="75d65ad774ada5d151590f644b58fe8f0f6f5d84cbc371b67e21ab54f707c00d"
Jul 16 22:31:11.546479 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:11.546292    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh_odh-ai-gateway-infra(33fe860a-b540-4945-b4d9-8d020d648494)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" podUID="33fe860a-b540-4945-b4d9-8d020d648494"
Jul 16 22:31:16.972665 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:16.972628    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"]
Jul 16 22:31:16.977298 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:16.977278    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"
Jul 16 22:31:17.005978 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:17.005873    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"]
Jul 16 22:31:17.092639 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:17.092596    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3381b9a0-6345-4637-b3bf-608b243d8bb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b\" (UID: \"3381b9a0-6345-4637-b3bf-608b243d8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"
Jul 16 22:31:17.193845 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:17.193803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3381b9a0-6345-4637-b3bf-608b243d8bb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b\" (UID: \"3381b9a0-6345-4637-b3bf-608b243d8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"
Jul 16 22:31:17.194493 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:17.194472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3381b9a0-6345-4637-b3bf-608b243d8bb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b\" (UID: \"3381b9a0-6345-4637-b3bf-608b243d8bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"
Jul 16 22:31:17.287860 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:17.287779    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"
Jul 16 22:31:17.468162 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:17.468135    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"]
Jul 16 22:31:17.470652 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:31:17.470626    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3381b9a0_6345_4637_b3bf_608b243d8bb1.slice/crio-193e885417f94b91532a691fc7cdbc2afef4a31e3e3792620b5c2e2ea9a94f49 WatchSource:0}: Error finding container 193e885417f94b91532a691fc7cdbc2afef4a31e3e3792620b5c2e2ea9a94f49: Status 404 returned error can't find the container with id 193e885417f94b91532a691fc7cdbc2afef4a31e3e3792620b5c2e2ea9a94f49
Jul 16 22:31:17.575421 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:17.575334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" event={"ID":"3381b9a0-6345-4637-b3bf-608b243d8bb1","Type":"ContainerStarted","Data":"193e885417f94b91532a691fc7cdbc2afef4a31e3e3792620b5c2e2ea9a94f49"}
Jul 16 22:31:18.582191 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:18.582159    2555 generic.go:356] "Generic (PLEG): container finished" podID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerID="0e1709dce7f01256a2a006705259a289c61b1793aad90869953947b5cd578588" exitCode=6
Jul 16 22:31:18.582571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:18.582209    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" event={"ID":"3381b9a0-6345-4637-b3bf-608b243d8bb1","Type":"ContainerDied","Data":"0e1709dce7f01256a2a006705259a289c61b1793aad90869953947b5cd578588"}
Jul 16 22:31:18.582571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:18.582378    2555 scope.go:117] "RemoveContainer" containerID="0e1709dce7f01256a2a006705259a289c61b1793aad90869953947b5cd578588"
Jul 16 22:31:19.588051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:19.588016    2555 generic.go:356] "Generic (PLEG): container finished" podID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerID="7d78d0718c7a22cd640d6bd482cb66d9038698ee3058b132ace3a5a432f12d47" exitCode=6
Jul 16 22:31:19.588490 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:19.588061    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" event={"ID":"3381b9a0-6345-4637-b3bf-608b243d8bb1","Type":"ContainerDied","Data":"7d78d0718c7a22cd640d6bd482cb66d9038698ee3058b132ace3a5a432f12d47"}
Jul 16 22:31:19.588490 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:19.588122    2555 scope.go:117] "RemoveContainer" containerID="0e1709dce7f01256a2a006705259a289c61b1793aad90869953947b5cd578588"
Jul 16 22:31:19.588490 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:19.588273    2555 scope.go:117] "RemoveContainer" containerID="7d78d0718c7a22cd640d6bd482cb66d9038698ee3058b132ace3a5a432f12d47"
Jul 16 22:31:19.588653 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:19.588516    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b_odh-ai-gateway-infra(3381b9a0-6345-4637-b3bf-608b243d8bb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1"
Jul 16 22:31:20.594285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:20.594253    2555 scope.go:117] "RemoveContainer" containerID="7d78d0718c7a22cd640d6bd482cb66d9038698ee3058b132ace3a5a432f12d47"
Jul 16 22:31:20.594681 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:20.594420    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b_odh-ai-gateway-infra(3381b9a0-6345-4637-b3bf-608b243d8bb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1"
Jul 16 22:31:24.491811 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:24.491781    2555 scope.go:117] "RemoveContainer" containerID="75d65ad774ada5d151590f644b58fe8f0f6f5d84cbc371b67e21ab54f707c00d"
Jul 16 22:31:25.618527 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:25.618497    2555 generic.go:356] "Generic (PLEG): container finished" podID="33fe860a-b540-4945-b4d9-8d020d648494" containerID="00c7f2a00b1e60e5a42e9ea6b2c139011b34e1f055ddb984e456d272db7c13d4" exitCode=6
Jul 16 22:31:25.618987 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:25.618574    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" event={"ID":"33fe860a-b540-4945-b4d9-8d020d648494","Type":"ContainerDied","Data":"00c7f2a00b1e60e5a42e9ea6b2c139011b34e1f055ddb984e456d272db7c13d4"}
Jul 16 22:31:25.618987 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:25.618652    2555 scope.go:117] "RemoveContainer" containerID="75d65ad774ada5d151590f644b58fe8f0f6f5d84cbc371b67e21ab54f707c00d"
Jul 16 22:31:25.619152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:25.619010    2555 scope.go:117] "RemoveContainer" containerID="00c7f2a00b1e60e5a42e9ea6b2c139011b34e1f055ddb984e456d272db7c13d4"
Jul 16 22:31:25.619246 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:25.619223    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh_odh-ai-gateway-infra(33fe860a-b540-4945-b4d9-8d020d648494)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" podUID="33fe860a-b540-4945-b4d9-8d020d648494"
Jul 16 22:31:26.666428 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:26.666393    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"]
Jul 16 22:31:26.795213 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:26.795182    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"
Jul 16 22:31:26.880236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:26.880193    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe860a-b540-4945-b4d9-8d020d648494-maas-api-service-ca\") pod \"33fe860a-b540-4945-b4d9-8d020d648494\" (UID: \"33fe860a-b540-4945-b4d9-8d020d648494\") "
Jul 16 22:31:26.880567 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:26.880542    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fe860a-b540-4945-b4d9-8d020d648494-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33fe860a-b540-4945-b4d9-8d020d648494" (UID: "33fe860a-b540-4945-b4d9-8d020d648494"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:26.981746 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:26.981647    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fe860a-b540-4945-b4d9-8d020d648494-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:31:27.628807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:27.628772    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"
Jul 16 22:31:27.628986 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:27.628777    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh" event={"ID":"33fe860a-b540-4945-b4d9-8d020d648494","Type":"ContainerDied","Data":"0aa137580a18e5d1d8de006e31a24daa97a358f96801b6ce022dbfbcc73ab4e6"}
Jul 16 22:31:27.628986 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:27.628902    2555 scope.go:117] "RemoveContainer" containerID="00c7f2a00b1e60e5a42e9ea6b2c139011b34e1f055ddb984e456d272db7c13d4"
Jul 16 22:31:27.669943 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:27.669906    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"]
Jul 16 22:31:27.680904 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:27.680869    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-xp5fh"]
Jul 16 22:31:28.494555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:28.494520    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33fe860a-b540-4945-b4d9-8d020d648494" path="/var/lib/kubelet/pods/33fe860a-b540-4945-b4d9-8d020d648494/volumes"
Jul 16 22:31:35.490011 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:35.489951    2555 scope.go:117] "RemoveContainer" containerID="7d78d0718c7a22cd640d6bd482cb66d9038698ee3058b132ace3a5a432f12d47"
Jul 16 22:31:36.668139 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:36.668100    2555 generic.go:356] "Generic (PLEG): container finished" podID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerID="d71d554d4bb80492f5043898168d8e5cfc589d5d1ba7d098ffa94e10d81bfee5" exitCode=6
Jul 16 22:31:36.668635 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:36.668146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" event={"ID":"3381b9a0-6345-4637-b3bf-608b243d8bb1","Type":"ContainerDied","Data":"d71d554d4bb80492f5043898168d8e5cfc589d5d1ba7d098ffa94e10d81bfee5"}
Jul 16 22:31:36.668635 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:36.668194    2555 scope.go:117] "RemoveContainer" containerID="7d78d0718c7a22cd640d6bd482cb66d9038698ee3058b132ace3a5a432f12d47"
Jul 16 22:31:36.668635 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:36.668509    2555 scope.go:117] "RemoveContainer" containerID="d71d554d4bb80492f5043898168d8e5cfc589d5d1ba7d098ffa94e10d81bfee5"
Jul 16 22:31:36.668815 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:36.668732    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b_odh-ai-gateway-infra(3381b9a0-6345-4637-b3bf-608b243d8bb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1"
Jul 16 22:31:37.741609 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:37.741573    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"]
Jul 16 22:31:37.873391 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:37.873340    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"
Jul 16 22:31:37.979355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:37.979320    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3381b9a0-6345-4637-b3bf-608b243d8bb1-maas-api-service-ca\") pod \"3381b9a0-6345-4637-b3bf-608b243d8bb1\" (UID: \"3381b9a0-6345-4637-b3bf-608b243d8bb1\") "
Jul 16 22:31:37.979660 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:37.979638    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3381b9a0-6345-4637-b3bf-608b243d8bb1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3381b9a0-6345-4637-b3bf-608b243d8bb1" (UID: "3381b9a0-6345-4637-b3bf-608b243d8bb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:38.080008 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:38.079946    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3381b9a0-6345-4637-b3bf-608b243d8bb1-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:31:38.679373 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:38.679329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b" event={"ID":"3381b9a0-6345-4637-b3bf-608b243d8bb1","Type":"ContainerDied","Data":"193e885417f94b91532a691fc7cdbc2afef4a31e3e3792620b5c2e2ea9a94f49"}
Jul 16 22:31:38.679373 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:38.679364    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"
Jul 16 22:31:38.679597 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:38.679405    2555 scope.go:117] "RemoveContainer" containerID="d71d554d4bb80492f5043898168d8e5cfc589d5d1ba7d098ffa94e10d81bfee5"
Jul 16 22:31:38.707405 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:38.707369    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"]
Jul 16 22:31:38.719658 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:38.719628    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zbr2b"]
Jul 16 22:31:40.494189 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:40.494159    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" path="/var/lib/kubelet/pods/3381b9a0-6345-4637-b3bf-608b243d8bb1/volumes"
Jul 16 22:31:58.354355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354324    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"]
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354621    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354631    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354642    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354647    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354654    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354660    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354667    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354672    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354682    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354687    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354733    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354740    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354746    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:31:58.354766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.354753    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:31:58.359206 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.359187    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:31:58.362727 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:31:58.362695    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-derive-67907cc2-s9qps\" is forbidden: User \"system:node:ip-10-0-143-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-143-124.ec2.internal' and this object" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:31:58.362872 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.362756    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:31:58.363523 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.363508    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:31:58.382259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.382237    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"]
Jul 16 22:31:58.459643 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.459596    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79a4398-7142-4988-b1ff-9b7af74eac93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-s9qps\" (UID: \"d79a4398-7142-4988-b1ff-9b7af74eac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:31:58.560162 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.560125    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79a4398-7142-4988-b1ff-9b7af74eac93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-s9qps\" (UID: \"d79a4398-7142-4988-b1ff-9b7af74eac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:31:58.560749 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.560728    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79a4398-7142-4988-b1ff-9b7af74eac93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-s9qps\" (UID: \"d79a4398-7142-4988-b1ff-9b7af74eac93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:31:58.668989 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.668863    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:31:58.859319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:58.859285    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"]
Jul 16 22:31:58.862530 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:31:58.862503    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79a4398_7142_4988_b1ff_9b7af74eac93.slice/crio-50e95f1f01cd0de8fe2c8d4b2484eb5123fc123c95156cc3dc7cdb04c105d42d WatchSource:0}: Error finding container 50e95f1f01cd0de8fe2c8d4b2484eb5123fc123c95156cc3dc7cdb04c105d42d: Status 404 returned error can't find the container with id 50e95f1f01cd0de8fe2c8d4b2484eb5123fc123c95156cc3dc7cdb04c105d42d
Jul 16 22:31:59.770721 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:59.770684    2555 generic.go:356] "Generic (PLEG): container finished" podID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerID="da0550163f0836a5af37b0e7147b4992258f0ce8efe7ea5e12687cdedfc3a2da" exitCode=6
Jul 16 22:31:59.771138 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:59.770756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" event={"ID":"d79a4398-7142-4988-b1ff-9b7af74eac93","Type":"ContainerDied","Data":"da0550163f0836a5af37b0e7147b4992258f0ce8efe7ea5e12687cdedfc3a2da"}
Jul 16 22:31:59.771138 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:59.770793    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" event={"ID":"d79a4398-7142-4988-b1ff-9b7af74eac93","Type":"ContainerStarted","Data":"50e95f1f01cd0de8fe2c8d4b2484eb5123fc123c95156cc3dc7cdb04c105d42d"}
Jul 16 22:31:59.771138 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:31:59.770931    2555 scope.go:117] "RemoveContainer" containerID="da0550163f0836a5af37b0e7147b4992258f0ce8efe7ea5e12687cdedfc3a2da"
Jul 16 22:32:00.777276 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:00.777182    2555 generic.go:356] "Generic (PLEG): container finished" podID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerID="ba6f9d98c6cad0bbb91132fa022ee79f603dee4151f36785bb4ef1dd3c967d56" exitCode=6
Jul 16 22:32:00.777276 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:00.777222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" event={"ID":"d79a4398-7142-4988-b1ff-9b7af74eac93","Type":"ContainerDied","Data":"ba6f9d98c6cad0bbb91132fa022ee79f603dee4151f36785bb4ef1dd3c967d56"}
Jul 16 22:32:00.777770 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:00.777287    2555 scope.go:117] "RemoveContainer" containerID="da0550163f0836a5af37b0e7147b4992258f0ce8efe7ea5e12687cdedfc3a2da"
Jul 16 22:32:00.777770 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:00.777452    2555 scope.go:117] "RemoveContainer" containerID="ba6f9d98c6cad0bbb91132fa022ee79f603dee4151f36785bb4ef1dd3c967d56"
Jul 16 22:32:00.777770 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:00.777660    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-s9qps_odh-ai-gateway-infra(d79a4398-7142-4988-b1ff-9b7af74eac93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93"
Jul 16 22:32:01.783143 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:01.783115    2555 scope.go:117] "RemoveContainer" containerID="ba6f9d98c6cad0bbb91132fa022ee79f603dee4151f36785bb4ef1dd3c967d56"
Jul 16 22:32:01.783557 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:01.783304    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-s9qps_odh-ai-gateway-infra(d79a4398-7142-4988-b1ff-9b7af74eac93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93"
Jul 16 22:32:12.489561 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:12.489527    2555 scope.go:117] "RemoveContainer" containerID="ba6f9d98c6cad0bbb91132fa022ee79f603dee4151f36785bb4ef1dd3c967d56"
Jul 16 22:32:12.831745 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:12.831708    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" event={"ID":"d79a4398-7142-4988-b1ff-9b7af74eac93","Type":"ContainerStarted","Data":"8c92d67ca558d8efe87f7f4279b9a3f148673bca5cc1b58be617b8e81ec15a5d"}
Jul 16 22:32:12.867611 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:12.867556    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" podStartSLOduration=14.867539187 podStartE2EDuration="14.867539187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:32:12.867350449 +0000 UTC m=+2688.860514603" watchObservedRunningTime="2026-07-16 22:32:12.867539187 +0000 UTC m=+2688.860703338"
Jul 16 22:32:13.837824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:13.837791    2555 generic.go:356] "Generic (PLEG): container finished" podID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerID="8c92d67ca558d8efe87f7f4279b9a3f148673bca5cc1b58be617b8e81ec15a5d" exitCode=6
Jul 16 22:32:13.838285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:13.837877    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" event={"ID":"d79a4398-7142-4988-b1ff-9b7af74eac93","Type":"ContainerDied","Data":"8c92d67ca558d8efe87f7f4279b9a3f148673bca5cc1b58be617b8e81ec15a5d"}
Jul 16 22:32:13.838285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:13.837943    2555 scope.go:117] "RemoveContainer" containerID="ba6f9d98c6cad0bbb91132fa022ee79f603dee4151f36785bb4ef1dd3c967d56"
Jul 16 22:32:13.838285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:13.838241    2555 scope.go:117] "RemoveContainer" containerID="8c92d67ca558d8efe87f7f4279b9a3f148673bca5cc1b58be617b8e81ec15a5d"
Jul 16 22:32:13.838483 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:13.838465    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-s9qps_odh-ai-gateway-infra(d79a4398-7142-4988-b1ff-9b7af74eac93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93"
Jul 16 22:32:13.890401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:13.890367    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"]
Jul 16 22:32:14.977562 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:14.977536    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:32:14.995532 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:14.995504    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79a4398-7142-4988-b1ff-9b7af74eac93-maas-api-service-ca\") pod \"d79a4398-7142-4988-b1ff-9b7af74eac93\" (UID: \"d79a4398-7142-4988-b1ff-9b7af74eac93\") "
Jul 16 22:32:14.995819 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:14.995796    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79a4398-7142-4988-b1ff-9b7af74eac93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d79a4398-7142-4988-b1ff-9b7af74eac93" (UID: "d79a4398-7142-4988-b1ff-9b7af74eac93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:15.096504 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:15.096423    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79a4398-7142-4988-b1ff-9b7af74eac93-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:32:15.849515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:15.849486    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"
Jul 16 22:32:15.849733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:15.849484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps" event={"ID":"d79a4398-7142-4988-b1ff-9b7af74eac93","Type":"ContainerDied","Data":"50e95f1f01cd0de8fe2c8d4b2484eb5123fc123c95156cc3dc7cdb04c105d42d"}
Jul 16 22:32:15.849733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:15.849626    2555 scope.go:117] "RemoveContainer" containerID="8c92d67ca558d8efe87f7f4279b9a3f148673bca5cc1b58be617b8e81ec15a5d"
Jul 16 22:32:15.874714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:15.874672    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"]
Jul 16 22:32:15.880040 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:15.880019    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s9qps"]
Jul 16 22:32:16.494542 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:16.494512    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" path="/var/lib/kubelet/pods/d79a4398-7142-4988-b1ff-9b7af74eac93/volumes"
Jul 16 22:32:17.108977 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.108928    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"]
Jul 16 22:32:17.109280 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109266    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:17.109325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109284    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:17.109325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109306    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:32:17.109325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109312    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:32:17.109325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109320    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:17.109325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109325    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:17.109472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109378    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:17.109472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109387    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fe860a-b540-4945-b4d9-8d020d648494" containerName="revoke-keys"
Jul 16 22:32:17.109472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109394    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:17.109472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.109401    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3381b9a0-6345-4637-b3bf-608b243d8bb1" containerName="revoke-keys"
Jul 16 22:32:17.113190 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.113172    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"
Jul 16 22:32:17.118700 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.118658    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:32:17.118865 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.118717    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:32:17.143462 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.143428    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"]
Jul 16 22:32:17.211207 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.211165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1428662-047e-4560-9bbe-bdb9591ef3dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh\" (UID: \"a1428662-047e-4560-9bbe-bdb9591ef3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"
Jul 16 22:32:17.312464 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.312427    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1428662-047e-4560-9bbe-bdb9591ef3dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh\" (UID: \"a1428662-047e-4560-9bbe-bdb9591ef3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"
Jul 16 22:32:17.313130 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.313110    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1428662-047e-4560-9bbe-bdb9591ef3dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh\" (UID: \"a1428662-047e-4560-9bbe-bdb9591ef3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"
Jul 16 22:32:17.428279 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.428180    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"
Jul 16 22:32:17.573565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.573539    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"]
Jul 16 22:32:17.574753 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:32:17.574724    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1428662_047e_4560_9bbe_bdb9591ef3dd.slice/crio-3150cfc7a5ea4972ad4eaa2af96ffd541ed4460aa9f40e68aa2eb96c3ba4f405 WatchSource:0}: Error finding container 3150cfc7a5ea4972ad4eaa2af96ffd541ed4460aa9f40e68aa2eb96c3ba4f405: Status 404 returned error can't find the container with id 3150cfc7a5ea4972ad4eaa2af96ffd541ed4460aa9f40e68aa2eb96c3ba4f405
Jul 16 22:32:17.860457 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.860417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" event={"ID":"a1428662-047e-4560-9bbe-bdb9591ef3dd","Type":"ContainerStarted","Data":"44ce17cf9871c037d41f80047faafeeae38fd893043857c95869da54ff1f117b"}
Jul 16 22:32:17.860648 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:17.860478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" event={"ID":"a1428662-047e-4560-9bbe-bdb9591ef3dd","Type":"ContainerStarted","Data":"3150cfc7a5ea4972ad4eaa2af96ffd541ed4460aa9f40e68aa2eb96c3ba4f405"}
Jul 16 22:32:18.865513 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:18.865477    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerID="44ce17cf9871c037d41f80047faafeeae38fd893043857c95869da54ff1f117b" exitCode=6
Jul 16 22:32:18.865946 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:18.865572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" event={"ID":"a1428662-047e-4560-9bbe-bdb9591ef3dd","Type":"ContainerDied","Data":"44ce17cf9871c037d41f80047faafeeae38fd893043857c95869da54ff1f117b"}
Jul 16 22:32:18.865946 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:18.865711    2555 scope.go:117] "RemoveContainer" containerID="44ce17cf9871c037d41f80047faafeeae38fd893043857c95869da54ff1f117b"
Jul 16 22:32:19.449455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.449297    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"]
Jul 16 22:32:19.449688 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.449669    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:19.449797 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.449691    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:19.449797 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.449793    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79a4398-7142-4988-b1ff-9b7af74eac93" containerName="revoke-keys"
Jul 16 22:32:19.452604 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.452587    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"
Jul 16 22:32:19.461371 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.461341    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"]
Jul 16 22:32:19.529632 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.529596    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f80ade-9945-468b-8163-f480458ad49b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-601103b4-m2z2n\" (UID: \"f5f80ade-9945-468b-8163-f480458ad49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"
Jul 16 22:32:19.630185 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.630148    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f80ade-9945-468b-8163-f480458ad49b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-601103b4-m2z2n\" (UID: \"f5f80ade-9945-468b-8163-f480458ad49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"
Jul 16 22:32:19.630753 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.630734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f80ade-9945-468b-8163-f480458ad49b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-601103b4-m2z2n\" (UID: \"f5f80ade-9945-468b-8163-f480458ad49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"
Jul 16 22:32:19.819491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.819407    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"
Jul 16 22:32:19.873918 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.873635    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerID="c427f13098791ba66c5886fa770217b69297bf7c75d2f68286d8f8dbfb901aa5" exitCode=6
Jul 16 22:32:19.873918 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.873674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" event={"ID":"a1428662-047e-4560-9bbe-bdb9591ef3dd","Type":"ContainerDied","Data":"c427f13098791ba66c5886fa770217b69297bf7c75d2f68286d8f8dbfb901aa5"}
Jul 16 22:32:19.873918 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.873769    2555 scope.go:117] "RemoveContainer" containerID="44ce17cf9871c037d41f80047faafeeae38fd893043857c95869da54ff1f117b"
Jul 16 22:32:19.874605 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.873938    2555 scope.go:117] "RemoveContainer" containerID="c427f13098791ba66c5886fa770217b69297bf7c75d2f68286d8f8dbfb901aa5"
Jul 16 22:32:19.874605 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:19.874184    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh_odh-ai-gateway-infra(a1428662-047e-4560-9bbe-bdb9591ef3dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd"
Jul 16 22:32:19.961010 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:19.960977    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"]
Jul 16 22:32:19.965052 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:32:19.965027    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f80ade_9945_468b_8163_f480458ad49b.slice/crio-110a5b5935d746681e8d330e31b123311fe0fb81da1360389923a329cf3443d3 WatchSource:0}: Error finding container 110a5b5935d746681e8d330e31b123311fe0fb81da1360389923a329cf3443d3: Status 404 returned error can't find the container with id 110a5b5935d746681e8d330e31b123311fe0fb81da1360389923a329cf3443d3
Jul 16 22:32:20.879964 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:20.879932    2555 scope.go:117] "RemoveContainer" containerID="c427f13098791ba66c5886fa770217b69297bf7c75d2f68286d8f8dbfb901aa5"
Jul 16 22:32:20.880439 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:20.880165    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh_odh-ai-gateway-infra(a1428662-047e-4560-9bbe-bdb9591ef3dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd"
Jul 16 22:32:20.881189 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:20.881166    2555 generic.go:356] "Generic (PLEG): container finished" podID="f5f80ade-9945-468b-8163-f480458ad49b" containerID="7eecd4628c185446f8b5c7a88850bea9a6662941bfe2fee52786a2f7cb99607b" exitCode=0
Jul 16 22:32:20.881325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:20.881240    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n" event={"ID":"f5f80ade-9945-468b-8163-f480458ad49b","Type":"ContainerDied","Data":"7eecd4628c185446f8b5c7a88850bea9a6662941bfe2fee52786a2f7cb99607b"}
Jul 16 22:32:20.881325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:20.881284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n" event={"ID":"f5f80ade-9945-468b-8163-f480458ad49b","Type":"ContainerStarted","Data":"110a5b5935d746681e8d330e31b123311fe0fb81da1360389923a329cf3443d3"}
Jul 16 22:32:22.002383 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:22.002357    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"
Jul 16 22:32:22.147811 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:22.147726    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f80ade-9945-468b-8163-f480458ad49b-maas-api-service-ca\") pod \"f5f80ade-9945-468b-8163-f480458ad49b\" (UID: \"f5f80ade-9945-468b-8163-f480458ad49b\") "
Jul 16 22:32:22.148115 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:22.148094    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f80ade-9945-468b-8163-f480458ad49b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5f80ade-9945-468b-8163-f480458ad49b" (UID: "f5f80ade-9945-468b-8163-f480458ad49b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:22.249182 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:22.249142    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f80ade-9945-468b-8163-f480458ad49b-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:32:22.890999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:22.890950    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n"
Jul 16 22:32:22.890999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:22.890992    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-601103b4-m2z2n" event={"ID":"f5f80ade-9945-468b-8163-f480458ad49b","Type":"ContainerDied","Data":"110a5b5935d746681e8d330e31b123311fe0fb81da1360389923a329cf3443d3"}
Jul 16 22:32:22.891217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:22.891042    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="110a5b5935d746681e8d330e31b123311fe0fb81da1360389923a329cf3443d3"
Jul 16 22:32:25.713701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.713667    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"]
Jul 16 22:32:25.714088 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.713987    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f80ade-9945-468b-8163-f480458ad49b" containerName="revoke-keys"
Jul 16 22:32:25.714088 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.714000    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f80ade-9945-468b-8163-f480458ad49b" containerName="revoke-keys"
Jul 16 22:32:25.714088 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.714063    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f80ade-9945-468b-8163-f480458ad49b" containerName="revoke-keys"
Jul 16 22:32:25.718023 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.718006    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"
Jul 16 22:32:25.726045 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.726020    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"]
Jul 16 22:32:25.776422 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.776382    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc8e52f-2934-4177-b7e2-8c6ed6fcc022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl\" (UID: \"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"
Jul 16 22:32:25.877673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.877635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc8e52f-2934-4177-b7e2-8c6ed6fcc022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl\" (UID: \"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"
Jul 16 22:32:25.878280 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:25.878262    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc8e52f-2934-4177-b7e2-8c6ed6fcc022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl\" (UID: \"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"
Jul 16 22:32:26.029127 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:26.029024    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"
Jul 16 22:32:26.158034 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:26.158009    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"]
Jul 16 22:32:26.160682 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:32:26.160650    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc8e52f_2934_4177_b7e2_8c6ed6fcc022.slice/crio-8a7926aac8c44b74bbe317e9f085f2cbaa0d6bb10325f1a0aabec16dc31fe527 WatchSource:0}: Error finding container 8a7926aac8c44b74bbe317e9f085f2cbaa0d6bb10325f1a0aabec16dc31fe527: Status 404 returned error can't find the container with id 8a7926aac8c44b74bbe317e9f085f2cbaa0d6bb10325f1a0aabec16dc31fe527
Jul 16 22:32:26.909848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:26.909818    2555 generic.go:356] "Generic (PLEG): container finished" podID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerID="a06d0175cc02fe2d4b437ac9f1e62bbb7f2e5cc3b3da428e8b4d2480b3f948f5" exitCode=6
Jul 16 22:32:26.910277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:26.909866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" event={"ID":"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022","Type":"ContainerDied","Data":"a06d0175cc02fe2d4b437ac9f1e62bbb7f2e5cc3b3da428e8b4d2480b3f948f5"}
Jul 16 22:32:26.910277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:26.909908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" event={"ID":"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022","Type":"ContainerStarted","Data":"8a7926aac8c44b74bbe317e9f085f2cbaa0d6bb10325f1a0aabec16dc31fe527"}
Jul 16 22:32:26.910277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:26.910125    2555 scope.go:117] "RemoveContainer" containerID="a06d0175cc02fe2d4b437ac9f1e62bbb7f2e5cc3b3da428e8b4d2480b3f948f5"
Jul 16 22:32:27.917356 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:27.917320    2555 generic.go:356] "Generic (PLEG): container finished" podID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerID="09deac299738efc10ef66898aca3e4990e5d7880accc5346c78280d3431c6c12" exitCode=6
Jul 16 22:32:27.917841 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:27.917364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" event={"ID":"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022","Type":"ContainerDied","Data":"09deac299738efc10ef66898aca3e4990e5d7880accc5346c78280d3431c6c12"}
Jul 16 22:32:27.917841 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:27.917426    2555 scope.go:117] "RemoveContainer" containerID="a06d0175cc02fe2d4b437ac9f1e62bbb7f2e5cc3b3da428e8b4d2480b3f948f5"
Jul 16 22:32:27.917841 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:27.917644    2555 scope.go:117] "RemoveContainer" containerID="09deac299738efc10ef66898aca3e4990e5d7880accc5346c78280d3431c6c12"
Jul 16 22:32:27.918054 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:27.917888    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl_odh-ai-gateway-infra(ebc8e52f-2934-4177-b7e2-8c6ed6fcc022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022"
Jul 16 22:32:28.924022 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:28.923995    2555 scope.go:117] "RemoveContainer" containerID="09deac299738efc10ef66898aca3e4990e5d7880accc5346c78280d3431c6c12"
Jul 16 22:32:28.924430 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:28.924161    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl_odh-ai-gateway-infra(ebc8e52f-2934-4177-b7e2-8c6ed6fcc022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022"
Jul 16 22:32:33.489798 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:33.489763    2555 scope.go:117] "RemoveContainer" containerID="c427f13098791ba66c5886fa770217b69297bf7c75d2f68286d8f8dbfb901aa5"
Jul 16 22:32:33.947781 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:33.947742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" event={"ID":"a1428662-047e-4560-9bbe-bdb9591ef3dd","Type":"ContainerStarted","Data":"4d927dac36fcb347b2e7e2153b8ef772fdbb3df5b244332132119841e9bf89bd"}
Jul 16 22:32:33.972939 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:33.972887    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" podStartSLOduration=16.972871658 podStartE2EDuration="16.972871658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:32:17.892608822 +0000 UTC m=+2693.885772985" watchObservedRunningTime="2026-07-16 22:32:33.972871658 +0000 UTC m=+2709.966035810"
Jul 16 22:32:34.953690 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:34.953651    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerID="4d927dac36fcb347b2e7e2153b8ef772fdbb3df5b244332132119841e9bf89bd" exitCode=6
Jul 16 22:32:34.954141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:34.953732    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" event={"ID":"a1428662-047e-4560-9bbe-bdb9591ef3dd","Type":"ContainerDied","Data":"4d927dac36fcb347b2e7e2153b8ef772fdbb3df5b244332132119841e9bf89bd"}
Jul 16 22:32:34.954141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:34.953800    2555 scope.go:117] "RemoveContainer" containerID="c427f13098791ba66c5886fa770217b69297bf7c75d2f68286d8f8dbfb901aa5"
Jul 16 22:32:34.954141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:34.954083    2555 scope.go:117] "RemoveContainer" containerID="4d927dac36fcb347b2e7e2153b8ef772fdbb3df5b244332132119841e9bf89bd"
Jul 16 22:32:34.954281 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:34.954264    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh_odh-ai-gateway-infra(a1428662-047e-4560-9bbe-bdb9591ef3dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd"
Jul 16 22:32:34.982405 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:34.982372    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"]
Jul 16 22:32:36.088332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.088304    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"
Jul 16 22:32:36.165507 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.165470    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1428662-047e-4560-9bbe-bdb9591ef3dd-maas-api-service-ca\") pod \"a1428662-047e-4560-9bbe-bdb9591ef3dd\" (UID: \"a1428662-047e-4560-9bbe-bdb9591ef3dd\") "
Jul 16 22:32:36.165887 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.165864    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1428662-047e-4560-9bbe-bdb9591ef3dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1428662-047e-4560-9bbe-bdb9591ef3dd" (UID: "a1428662-047e-4560-9bbe-bdb9591ef3dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:36.267103 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.267019    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1428662-047e-4560-9bbe-bdb9591ef3dd-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:32:36.964678 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.964641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh" event={"ID":"a1428662-047e-4560-9bbe-bdb9591ef3dd","Type":"ContainerDied","Data":"3150cfc7a5ea4972ad4eaa2af96ffd541ed4460aa9f40e68aa2eb96c3ba4f405"}
Jul 16 22:32:36.964678 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.964678    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"
Jul 16 22:32:36.964942 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.964712    2555 scope.go:117] "RemoveContainer" containerID="4d927dac36fcb347b2e7e2153b8ef772fdbb3df5b244332132119841e9bf89bd"
Jul 16 22:32:36.982530 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.982502    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"]
Jul 16 22:32:36.985398 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:36.985377    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7v2wh"]
Jul 16 22:32:38.495185 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:38.495153    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" path="/var/lib/kubelet/pods/a1428662-047e-4560-9bbe-bdb9591ef3dd/volumes"
Jul 16 22:32:44.491881 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:44.491850    2555 scope.go:117] "RemoveContainer" containerID="09deac299738efc10ef66898aca3e4990e5d7880accc5346c78280d3431c6c12"
Jul 16 22:32:44.999338 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:44.999258    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" event={"ID":"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022","Type":"ContainerStarted","Data":"61fc9fe8cf369a75220af5868f9aa058d9605f773a07edeaa82975b844501925"}
Jul 16 22:32:45.019672 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:45.019621    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" podStartSLOduration=20.019607431 podStartE2EDuration="20.019607431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:32:45.018518405 +0000 UTC m=+2721.011682561" watchObservedRunningTime="2026-07-16 22:32:45.019607431 +0000 UTC m=+2721.012771588"
Jul 16 22:32:46.005509 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:46.005475    2555 generic.go:356] "Generic (PLEG): container finished" podID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerID="61fc9fe8cf369a75220af5868f9aa058d9605f773a07edeaa82975b844501925" exitCode=6
Jul 16 22:32:46.005949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:46.005524    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" event={"ID":"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022","Type":"ContainerDied","Data":"61fc9fe8cf369a75220af5868f9aa058d9605f773a07edeaa82975b844501925"}
Jul 16 22:32:46.005949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:46.005577    2555 scope.go:117] "RemoveContainer" containerID="09deac299738efc10ef66898aca3e4990e5d7880accc5346c78280d3431c6c12"
Jul 16 22:32:46.005949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:46.005888    2555 scope.go:117] "RemoveContainer" containerID="61fc9fe8cf369a75220af5868f9aa058d9605f773a07edeaa82975b844501925"
Jul 16 22:32:46.006148 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:46.006128    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl_odh-ai-gateway-infra(ebc8e52f-2934-4177-b7e2-8c6ed6fcc022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022"
Jul 16 22:32:46.035215 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:46.035190    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"]
Jul 16 22:32:47.138162 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:47.138139    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"
Jul 16 22:32:47.260066 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:47.260037    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc8e52f-2934-4177-b7e2-8c6ed6fcc022-maas-api-service-ca\") pod \"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022\" (UID: \"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022\") "
Jul 16 22:32:47.260495 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:47.260466    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc8e52f-2934-4177-b7e2-8c6ed6fcc022-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" (UID: "ebc8e52f-2934-4177-b7e2-8c6ed6fcc022"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:47.360707 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:47.360670    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc8e52f-2934-4177-b7e2-8c6ed6fcc022-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:32:48.016838 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.016797    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl" event={"ID":"ebc8e52f-2934-4177-b7e2-8c6ed6fcc022","Type":"ContainerDied","Data":"8a7926aac8c44b74bbe317e9f085f2cbaa0d6bb10325f1a0aabec16dc31fe527"}
Jul 16 22:32:48.016838 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.016839    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"
Jul 16 22:32:48.017112 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.016863    2555 scope.go:117] "RemoveContainer" containerID="61fc9fe8cf369a75220af5868f9aa058d9605f773a07edeaa82975b844501925"
Jul 16 22:32:48.042415 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.042385    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"]
Jul 16 22:32:48.050098 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.050072    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x4lwl"]
Jul 16 22:32:48.387456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387417    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"]
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387720    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387732    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387742    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387750    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387767    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387773    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387781    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387786    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387800    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.387829 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387805    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.388177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387870    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:32:48.388177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387880    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.388177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387886    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:32:48.388177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387892    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.388177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.387901    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1428662-047e-4560-9bbe-bdb9591ef3dd" containerName="revoke-keys"
Jul 16 22:32:48.392257 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.392240    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"
Jul 16 22:32:48.395094 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.395071    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:32:48.395094 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.395073    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:32:48.412303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.412275    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"]
Jul 16 22:32:48.470944 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.470902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633485d-2d75-4c26-8e28-7a49502ff6b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg\" (UID: \"8633485d-2d75-4c26-8e28-7a49502ff6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"
Jul 16 22:32:48.495766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.495731    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" path="/var/lib/kubelet/pods/ebc8e52f-2934-4177-b7e2-8c6ed6fcc022/volumes"
Jul 16 22:32:48.571502 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.571466    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633485d-2d75-4c26-8e28-7a49502ff6b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg\" (UID: \"8633485d-2d75-4c26-8e28-7a49502ff6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"
Jul 16 22:32:48.572163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.572145    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633485d-2d75-4c26-8e28-7a49502ff6b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg\" (UID: \"8633485d-2d75-4c26-8e28-7a49502ff6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"
Jul 16 22:32:48.702145 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.702056    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"
Jul 16 22:32:48.829599 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:48.829577    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"]
Jul 16 22:32:48.831069 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:32:48.831036    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8633485d_2d75_4c26_8e28_7a49502ff6b2.slice/crio-0b140541f73c46cd50b6193b78a4b318dad96fa0afaa1d913d53fb24a9c75d3c WatchSource:0}: Error finding container 0b140541f73c46cd50b6193b78a4b318dad96fa0afaa1d913d53fb24a9c75d3c: Status 404 returned error can't find the container with id 0b140541f73c46cd50b6193b78a4b318dad96fa0afaa1d913d53fb24a9c75d3c
Jul 16 22:32:49.022603 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:49.022516    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" event={"ID":"8633485d-2d75-4c26-8e28-7a49502ff6b2","Type":"ContainerStarted","Data":"0b140541f73c46cd50b6193b78a4b318dad96fa0afaa1d913d53fb24a9c75d3c"}
Jul 16 22:32:50.031684 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:50.031649    2555 generic.go:356] "Generic (PLEG): container finished" podID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerID="6bdeeb9c27518ac9565f8ae2d87e9ccf4eaad3f3561c947fd97af4db6ad1dff4" exitCode=6
Jul 16 22:32:50.032144 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:50.031718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" event={"ID":"8633485d-2d75-4c26-8e28-7a49502ff6b2","Type":"ContainerDied","Data":"6bdeeb9c27518ac9565f8ae2d87e9ccf4eaad3f3561c947fd97af4db6ad1dff4"}
Jul 16 22:32:50.032144 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:50.031905    2555 scope.go:117] "RemoveContainer" containerID="6bdeeb9c27518ac9565f8ae2d87e9ccf4eaad3f3561c947fd97af4db6ad1dff4"
Jul 16 22:32:51.038419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:51.038327    2555 generic.go:356] "Generic (PLEG): container finished" podID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerID="49d4aca08172605f889dfe7af7f1825fe3a5c05f5971ecd85b332cad83200fa1" exitCode=6
Jul 16 22:32:51.038837 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:51.038415    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" event={"ID":"8633485d-2d75-4c26-8e28-7a49502ff6b2","Type":"ContainerDied","Data":"49d4aca08172605f889dfe7af7f1825fe3a5c05f5971ecd85b332cad83200fa1"}
Jul 16 22:32:51.038837 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:51.038478    2555 scope.go:117] "RemoveContainer" containerID="6bdeeb9c27518ac9565f8ae2d87e9ccf4eaad3f3561c947fd97af4db6ad1dff4"
Jul 16 22:32:51.038837 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:51.038756    2555 scope.go:117] "RemoveContainer" containerID="49d4aca08172605f889dfe7af7f1825fe3a5c05f5971ecd85b332cad83200fa1"
Jul 16 22:32:51.039034 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:51.038998    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg_odh-ai-gateway-infra(8633485d-2d75-4c26-8e28-7a49502ff6b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2"
Jul 16 22:32:52.044372 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:32:52.044337    2555 scope.go:117] "RemoveContainer" containerID="49d4aca08172605f889dfe7af7f1825fe3a5c05f5971ecd85b332cad83200fa1"
Jul 16 22:32:52.044767 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:32:52.044520    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg_odh-ai-gateway-infra(8633485d-2d75-4c26-8e28-7a49502ff6b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2"
Jul 16 22:33:04.491337 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:04.491306    2555 scope.go:117] "RemoveContainer" containerID="49d4aca08172605f889dfe7af7f1825fe3a5c05f5971ecd85b332cad83200fa1"
Jul 16 22:33:05.097103 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:05.097068    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" event={"ID":"8633485d-2d75-4c26-8e28-7a49502ff6b2","Type":"ContainerStarted","Data":"548ab580f7a9eb89f621b857605876489bafab259c85d17b17dc8b13a60db844"}
Jul 16 22:33:05.115369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:05.115318    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" podStartSLOduration=17.115302404 podStartE2EDuration="17.115302404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:05.113631341 +0000 UTC m=+2741.106795484" watchObservedRunningTime="2026-07-16 22:33:05.115302404 +0000 UTC m=+2741.108466555"
Jul 16 22:33:06.103654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:06.103617    2555 generic.go:356] "Generic (PLEG): container finished" podID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerID="548ab580f7a9eb89f621b857605876489bafab259c85d17b17dc8b13a60db844" exitCode=6
Jul 16 22:33:06.104119 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:06.103695    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" event={"ID":"8633485d-2d75-4c26-8e28-7a49502ff6b2","Type":"ContainerDied","Data":"548ab580f7a9eb89f621b857605876489bafab259c85d17b17dc8b13a60db844"}
Jul 16 22:33:06.104119 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:06.103772    2555 scope.go:117] "RemoveContainer" containerID="49d4aca08172605f889dfe7af7f1825fe3a5c05f5971ecd85b332cad83200fa1"
Jul 16 22:33:06.104119 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:06.104106    2555 scope.go:117] "RemoveContainer" containerID="548ab580f7a9eb89f621b857605876489bafab259c85d17b17dc8b13a60db844"
Jul 16 22:33:06.104325 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:33:06.104307    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg_odh-ai-gateway-infra(8633485d-2d75-4c26-8e28-7a49502ff6b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2"
Jul 16 22:33:06.132220 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:06.132188    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"]
Jul 16 22:33:07.105118 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.105088    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"]
Jul 16 22:33:07.105614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.105405    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:33:07.105614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.105419    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:33:07.105614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.105498    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc8e52f-2934-4177-b7e2-8c6ed6fcc022" containerName="revoke-keys"
Jul 16 22:33:07.109861 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.109830    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"
Jul 16 22:33:07.116848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.116800    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"]
Jul 16 22:33:07.232373 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.232346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864dcd2f-c1bf-4e0b-965a-a9079d3266bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh\" (UID: \"864dcd2f-c1bf-4e0b-965a-a9079d3266bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"
Jul 16 22:33:07.243908 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.243888    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"
Jul 16 22:33:07.333690 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.333646    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633485d-2d75-4c26-8e28-7a49502ff6b2-maas-api-service-ca\") pod \"8633485d-2d75-4c26-8e28-7a49502ff6b2\" (UID: \"8633485d-2d75-4c26-8e28-7a49502ff6b2\") "
Jul 16 22:33:07.333944 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.333922    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864dcd2f-c1bf-4e0b-965a-a9079d3266bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh\" (UID: \"864dcd2f-c1bf-4e0b-965a-a9079d3266bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"
Jul 16 22:33:07.334065 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.334022    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8633485d-2d75-4c26-8e28-7a49502ff6b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8633485d-2d75-4c26-8e28-7a49502ff6b2" (UID: "8633485d-2d75-4c26-8e28-7a49502ff6b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:07.334572 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.334554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864dcd2f-c1bf-4e0b-965a-a9079d3266bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh\" (UID: \"864dcd2f-c1bf-4e0b-965a-a9079d3266bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"
Jul 16 22:33:07.423982 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.423858    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"
Jul 16 22:33:07.435243 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.435218    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633485d-2d75-4c26-8e28-7a49502ff6b2-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:33:07.555403 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:07.555381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"]
Jul 16 22:33:07.558116 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:33:07.558094    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864dcd2f_c1bf_4e0b_965a_a9079d3266bd.slice/crio-4bf7388ef33544f8546d65c44a9366b38aaf29ebda861f68aa4a42cf69a45cea WatchSource:0}: Error finding container 4bf7388ef33544f8546d65c44a9366b38aaf29ebda861f68aa4a42cf69a45cea: Status 404 returned error can't find the container with id 4bf7388ef33544f8546d65c44a9366b38aaf29ebda861f68aa4a42cf69a45cea
Jul 16 22:33:08.115633 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.115598    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"
Jul 16 22:33:08.115633 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.115619    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg" event={"ID":"8633485d-2d75-4c26-8e28-7a49502ff6b2","Type":"ContainerDied","Data":"0b140541f73c46cd50b6193b78a4b318dad96fa0afaa1d913d53fb24a9c75d3c"}
Jul 16 22:33:08.116169 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.115701    2555 scope.go:117] "RemoveContainer" containerID="548ab580f7a9eb89f621b857605876489bafab259c85d17b17dc8b13a60db844"
Jul 16 22:33:08.117305 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.117276    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" event={"ID":"864dcd2f-c1bf-4e0b-965a-a9079d3266bd","Type":"ContainerStarted","Data":"566badcd316d17425fbcc8ab79b5c9095527ed311ad11babd0ad53b899342c00"}
Jul 16 22:33:08.117443 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.117333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" event={"ID":"864dcd2f-c1bf-4e0b-965a-a9079d3266bd","Type":"ContainerStarted","Data":"4bf7388ef33544f8546d65c44a9366b38aaf29ebda861f68aa4a42cf69a45cea"}
Jul 16 22:33:08.140803 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.140750    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" podStartSLOduration=1.140731737 podStartE2EDuration="1.140731737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:08.139389734 +0000 UTC m=+2744.132553886" watchObservedRunningTime="2026-07-16 22:33:08.140731737 +0000 UTC m=+2744.133895889"
Jul 16 22:33:08.157906 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.157878    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"]
Jul 16 22:33:08.164649 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.164625    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hl9sg"]
Jul 16 22:33:08.494483 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:08.494400    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" path="/var/lib/kubelet/pods/8633485d-2d75-4c26-8e28-7a49502ff6b2/volumes"
Jul 16 22:33:09.123742 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:09.123705    2555 generic.go:356] "Generic (PLEG): container finished" podID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerID="566badcd316d17425fbcc8ab79b5c9095527ed311ad11babd0ad53b899342c00" exitCode=6
Jul 16 22:33:09.124239 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:09.123800    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" event={"ID":"864dcd2f-c1bf-4e0b-965a-a9079d3266bd","Type":"ContainerDied","Data":"566badcd316d17425fbcc8ab79b5c9095527ed311ad11babd0ad53b899342c00"}
Jul 16 22:33:09.124239 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:09.124039    2555 scope.go:117] "RemoveContainer" containerID="566badcd316d17425fbcc8ab79b5c9095527ed311ad11babd0ad53b899342c00"
Jul 16 22:33:10.130566 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:10.130531    2555 generic.go:356] "Generic (PLEG): container finished" podID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerID="7b205322789dfe764735604596c0e88b2cc9304c4153e501a6b9989dc30275ad" exitCode=6
Jul 16 22:33:10.130566 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:10.130570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" event={"ID":"864dcd2f-c1bf-4e0b-965a-a9079d3266bd","Type":"ContainerDied","Data":"7b205322789dfe764735604596c0e88b2cc9304c4153e501a6b9989dc30275ad"}
Jul 16 22:33:10.131133 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:10.130612    2555 scope.go:117] "RemoveContainer" containerID="566badcd316d17425fbcc8ab79b5c9095527ed311ad11babd0ad53b899342c00"
Jul 16 22:33:10.131133 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:10.130783    2555 scope.go:117] "RemoveContainer" containerID="7b205322789dfe764735604596c0e88b2cc9304c4153e501a6b9989dc30275ad"
Jul 16 22:33:10.131133 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:33:10.131029    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh_odh-ai-gateway-infra(864dcd2f-c1bf-4e0b-965a-a9079d3266bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd"
Jul 16 22:33:11.136583 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:11.136560    2555 scope.go:117] "RemoveContainer" containerID="7b205322789dfe764735604596c0e88b2cc9304c4153e501a6b9989dc30275ad"
Jul 16 22:33:11.136952 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:33:11.136769    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh_odh-ai-gateway-infra(864dcd2f-c1bf-4e0b-965a-a9079d3266bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd"
Jul 16 22:33:21.489307 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:21.489277    2555 scope.go:117] "RemoveContainer" containerID="7b205322789dfe764735604596c0e88b2cc9304c4153e501a6b9989dc30275ad"
Jul 16 22:33:22.184866 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:22.184834    2555 generic.go:356] "Generic (PLEG): container finished" podID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerID="1f850db7904654c72a385d00cdb74ddc74216aedefef303a82d1c392ac86422d" exitCode=6
Jul 16 22:33:22.185053 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:22.184898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" event={"ID":"864dcd2f-c1bf-4e0b-965a-a9079d3266bd","Type":"ContainerDied","Data":"1f850db7904654c72a385d00cdb74ddc74216aedefef303a82d1c392ac86422d"}
Jul 16 22:33:22.185053 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:22.185005    2555 scope.go:117] "RemoveContainer" containerID="7b205322789dfe764735604596c0e88b2cc9304c4153e501a6b9989dc30275ad"
Jul 16 22:33:22.185271 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:22.185253    2555 scope.go:117] "RemoveContainer" containerID="1f850db7904654c72a385d00cdb74ddc74216aedefef303a82d1c392ac86422d"
Jul 16 22:33:22.185503 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:33:22.185469    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh_odh-ai-gateway-infra(864dcd2f-c1bf-4e0b-965a-a9079d3266bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd"
Jul 16 22:33:23.213298 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:23.213256    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"]
Jul 16 22:33:23.343525 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:23.343501    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"
Jul 16 22:33:23.365463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:23.365432    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864dcd2f-c1bf-4e0b-965a-a9079d3266bd-maas-api-service-ca\") pod \"864dcd2f-c1bf-4e0b-965a-a9079d3266bd\" (UID: \"864dcd2f-c1bf-4e0b-965a-a9079d3266bd\") "
Jul 16 22:33:23.365872 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:23.365849    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864dcd2f-c1bf-4e0b-965a-a9079d3266bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "864dcd2f-c1bf-4e0b-965a-a9079d3266bd" (UID: "864dcd2f-c1bf-4e0b-965a-a9079d3266bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:23.466590 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:23.466510    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/864dcd2f-c1bf-4e0b-965a-a9079d3266bd-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:33:24.195789 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:24.195756    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"
Jul 16 22:33:24.195789 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:24.195775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh" event={"ID":"864dcd2f-c1bf-4e0b-965a-a9079d3266bd","Type":"ContainerDied","Data":"4bf7388ef33544f8546d65c44a9366b38aaf29ebda861f68aa4a42cf69a45cea"}
Jul 16 22:33:24.196077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:24.195837    2555 scope.go:117] "RemoveContainer" containerID="1f850db7904654c72a385d00cdb74ddc74216aedefef303a82d1c392ac86422d"
Jul 16 22:33:24.217849 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:24.217823    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"]
Jul 16 22:33:24.224576 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:24.224552    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-tpvqh"]
Jul 16 22:33:24.495277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:24.495198    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" path="/var/lib/kubelet/pods/864dcd2f-c1bf-4e0b-965a-a9079d3266bd/volumes"
Jul 16 22:33:25.752926 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.752893    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"]
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753200    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753211    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753223    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753228    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753240    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753245    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753253    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753258    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753265    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753270    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753276    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753282    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753331    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753338    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753342    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.753755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753348    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:33:25.753755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.753353    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="864dcd2f-c1bf-4e0b-965a-a9079d3266bd" containerName="revoke-keys"
Jul 16 22:33:25.761896 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.761872    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"
Jul 16 22:33:25.763316 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.763294    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"]
Jul 16 22:33:25.764690 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.764666    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:25.764805 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.764729    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:33:25.787284 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.787258    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bacadc-d4e1-4700-8624-fe6946c73894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh\" (UID: \"a8bacadc-d4e1-4700-8624-fe6946c73894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"
Jul 16 22:33:25.887942 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.887886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bacadc-d4e1-4700-8624-fe6946c73894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh\" (UID: \"a8bacadc-d4e1-4700-8624-fe6946c73894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"
Jul 16 22:33:25.888547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:25.888526    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bacadc-d4e1-4700-8624-fe6946c73894-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh\" (UID: \"a8bacadc-d4e1-4700-8624-fe6946c73894\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"
Jul 16 22:33:26.073244 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:26.073148    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"
Jul 16 22:33:26.227591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:26.227560    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"]
Jul 16 22:33:26.228140 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:33:26.228118    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bacadc_d4e1_4700_8624_fe6946c73894.slice/crio-f0cc6b96ec20aae8fab096e2d581237c23bbe58a4a583b900bf2476ef383a253 WatchSource:0}: Error finding container f0cc6b96ec20aae8fab096e2d581237c23bbe58a4a583b900bf2476ef383a253: Status 404 returned error can't find the container with id f0cc6b96ec20aae8fab096e2d581237c23bbe58a4a583b900bf2476ef383a253
Jul 16 22:33:27.210444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:27.210356    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerID="232a97e5ab561115dce698d88e1607b497c584735ce2b6c7267f1f85c30ba93e" exitCode=6
Jul 16 22:33:27.210444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:27.210394    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" event={"ID":"a8bacadc-d4e1-4700-8624-fe6946c73894","Type":"ContainerDied","Data":"232a97e5ab561115dce698d88e1607b497c584735ce2b6c7267f1f85c30ba93e"}
Jul 16 22:33:27.210444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:27.210443    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" event={"ID":"a8bacadc-d4e1-4700-8624-fe6946c73894","Type":"ContainerStarted","Data":"f0cc6b96ec20aae8fab096e2d581237c23bbe58a4a583b900bf2476ef383a253"}
Jul 16 22:33:27.211031 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:27.210679    2555 scope.go:117] "RemoveContainer" containerID="232a97e5ab561115dce698d88e1607b497c584735ce2b6c7267f1f85c30ba93e"
Jul 16 22:33:28.217392 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:28.217297    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerID="e662e14d92ad3952e4de3b5ca1faef59adb2ab7cf6c0ef074db64043c91a0aa1" exitCode=6
Jul 16 22:33:28.217392 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:28.217348    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" event={"ID":"a8bacadc-d4e1-4700-8624-fe6946c73894","Type":"ContainerDied","Data":"e662e14d92ad3952e4de3b5ca1faef59adb2ab7cf6c0ef074db64043c91a0aa1"}
Jul 16 22:33:28.218003 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:28.217407    2555 scope.go:117] "RemoveContainer" containerID="232a97e5ab561115dce698d88e1607b497c584735ce2b6c7267f1f85c30ba93e"
Jul 16 22:33:28.218003 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:28.217605    2555 scope.go:117] "RemoveContainer" containerID="e662e14d92ad3952e4de3b5ca1faef59adb2ab7cf6c0ef074db64043c91a0aa1"
Jul 16 22:33:28.218003 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:33:28.217833    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh_odh-ai-gateway-infra(a8bacadc-d4e1-4700-8624-fe6946c73894)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894"
Jul 16 22:33:29.225294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:29.225270    2555 scope.go:117] "RemoveContainer" containerID="e662e14d92ad3952e4de3b5ca1faef59adb2ab7cf6c0ef074db64043c91a0aa1"
Jul 16 22:33:29.225676 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:33:29.225452    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh_odh-ai-gateway-infra(a8bacadc-d4e1-4700-8624-fe6946c73894)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894"
Jul 16 22:33:43.490105 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:43.490072    2555 scope.go:117] "RemoveContainer" containerID="e662e14d92ad3952e4de3b5ca1faef59adb2ab7cf6c0ef074db64043c91a0aa1"
Jul 16 22:33:44.297749 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:44.297646    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerID="258301f99f533bc2033071b6a1c1985711632e00519ea50789f46ab3b46f893a" exitCode=6
Jul 16 22:33:44.297905 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:44.297698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" event={"ID":"a8bacadc-d4e1-4700-8624-fe6946c73894","Type":"ContainerDied","Data":"258301f99f533bc2033071b6a1c1985711632e00519ea50789f46ab3b46f893a"}
Jul 16 22:33:44.297905 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:44.297853    2555 scope.go:117] "RemoveContainer" containerID="e662e14d92ad3952e4de3b5ca1faef59adb2ab7cf6c0ef074db64043c91a0aa1"
Jul 16 22:33:44.298179 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:44.298162    2555 scope.go:117] "RemoveContainer" containerID="258301f99f533bc2033071b6a1c1985711632e00519ea50789f46ab3b46f893a"
Jul 16 22:33:44.298389 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:33:44.298369    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh_odh-ai-gateway-infra(a8bacadc-d4e1-4700-8624-fe6946c73894)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894"
Jul 16 22:33:45.343575 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:45.343537    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"]
Jul 16 22:33:45.471170 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:45.471146    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"
Jul 16 22:33:45.553361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:45.553272    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bacadc-d4e1-4700-8624-fe6946c73894-maas-api-service-ca\") pod \"a8bacadc-d4e1-4700-8624-fe6946c73894\" (UID: \"a8bacadc-d4e1-4700-8624-fe6946c73894\") "
Jul 16 22:33:45.553700 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:45.553670    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bacadc-d4e1-4700-8624-fe6946c73894-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8bacadc-d4e1-4700-8624-fe6946c73894" (UID: "a8bacadc-d4e1-4700-8624-fe6946c73894"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:45.654688 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:45.654604    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bacadc-d4e1-4700-8624-fe6946c73894-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:33:46.308331 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:46.308299    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"
Jul 16 22:33:46.308512 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:46.308299    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh" event={"ID":"a8bacadc-d4e1-4700-8624-fe6946c73894","Type":"ContainerDied","Data":"f0cc6b96ec20aae8fab096e2d581237c23bbe58a4a583b900bf2476ef383a253"}
Jul 16 22:33:46.308512 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:46.308435    2555 scope.go:117] "RemoveContainer" containerID="258301f99f533bc2033071b6a1c1985711632e00519ea50789f46ab3b46f893a"
Jul 16 22:33:46.332297 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:46.332268    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"]
Jul 16 22:33:46.337209 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:46.337187    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h8cdh"]
Jul 16 22:33:46.494879 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:33:46.494845    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" path="/var/lib/kubelet/pods/a8bacadc-d4e1-4700-8624-fe6946c73894/volumes"
Jul 16 22:34:25.479134 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479100    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"]
Jul 16 22:34:25.479547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479437    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.479547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479450    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.479547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479458    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.479547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479463    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.479547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479513    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8633485d-2d75-4c26-8e28-7a49502ff6b2" containerName="revoke-keys"
Jul 16 22:34:25.479547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479524    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.479547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.479530    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.484015 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.483998    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"
Jul 16 22:34:25.486775 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.486755    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:34:25.486895 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.486847    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:34:25.492243 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.492219    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"]
Jul 16 22:34:25.602275 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.602238    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ca8bc-ae65-4527-b803-7f99a1f7a431-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s\" (UID: \"c43ca8bc-ae65-4527-b803-7f99a1f7a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"
Jul 16 22:34:25.703769 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.703727    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ca8bc-ae65-4527-b803-7f99a1f7a431-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s\" (UID: \"c43ca8bc-ae65-4527-b803-7f99a1f7a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"
Jul 16 22:34:25.704419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.704401    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ca8bc-ae65-4527-b803-7f99a1f7a431-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s\" (UID: \"c43ca8bc-ae65-4527-b803-7f99a1f7a431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"
Jul 16 22:34:25.795809 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.795721    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"
Jul 16 22:34:25.899828 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.899800    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"]
Jul 16 22:34:25.900185 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.900171    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.900234 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.900187    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.900272 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.900240    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bacadc-d4e1-4700-8624-fe6946c73894" containerName="revoke-keys"
Jul 16 22:34:25.904449 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.904428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"
Jul 16 22:34:25.919481 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.919446    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"]
Jul 16 22:34:25.941000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:25.940976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"]
Jul 16 22:34:25.942539 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:34:25.942501    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43ca8bc_ae65_4527_b803_7f99a1f7a431.slice/crio-c02842a19261ca0f5c3dd7f1db8d8cca9682bb5388bfcc2c0d62fa023865ea33 WatchSource:0}: Error finding container c02842a19261ca0f5c3dd7f1db8d8cca9682bb5388bfcc2c0d62fa023865ea33: Status 404 returned error can't find the container with id c02842a19261ca0f5c3dd7f1db8d8cca9682bb5388bfcc2c0d62fa023865ea33
Jul 16 22:34:26.006141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.006107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/966175f6-c81e-484c-ad10-0bb3a0b7a595-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7\" (UID: \"966175f6-c81e-484c-ad10-0bb3a0b7a595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"
Jul 16 22:34:26.107182 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.107147    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/966175f6-c81e-484c-ad10-0bb3a0b7a595-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7\" (UID: \"966175f6-c81e-484c-ad10-0bb3a0b7a595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"
Jul 16 22:34:26.107766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.107747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/966175f6-c81e-484c-ad10-0bb3a0b7a595-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7\" (UID: \"966175f6-c81e-484c-ad10-0bb3a0b7a595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"
Jul 16 22:34:26.221848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.221802    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"
Jul 16 22:34:26.362007 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.361983    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"]
Jul 16 22:34:26.363723 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:34:26.363695    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966175f6_c81e_484c_ad10_0bb3a0b7a595.slice/crio-b14e2940bb4da2d2652df1a7cece6bae09c628217a6d05c5a4ffa7ad1762737d WatchSource:0}: Error finding container b14e2940bb4da2d2652df1a7cece6bae09c628217a6d05c5a4ffa7ad1762737d: Status 404 returned error can't find the container with id b14e2940bb4da2d2652df1a7cece6bae09c628217a6d05c5a4ffa7ad1762737d
Jul 16 22:34:26.481768 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.481734    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" event={"ID":"966175f6-c81e-484c-ad10-0bb3a0b7a595","Type":"ContainerStarted","Data":"b14e2940bb4da2d2652df1a7cece6bae09c628217a6d05c5a4ffa7ad1762737d"}
Jul 16 22:34:26.483137 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.483111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" event={"ID":"c43ca8bc-ae65-4527-b803-7f99a1f7a431","Type":"ContainerStarted","Data":"edf6692017bd2ad4e3635ab5b378bb22e59f4e5a11eac2832e6dab17e040b7a6"}
Jul 16 22:34:26.483328 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.483157    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" event={"ID":"c43ca8bc-ae65-4527-b803-7f99a1f7a431","Type":"ContainerStarted","Data":"c02842a19261ca0f5c3dd7f1db8d8cca9682bb5388bfcc2c0d62fa023865ea33"}
Jul 16 22:34:26.503290 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:26.503242    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" podStartSLOduration=1.503226567 podStartE2EDuration="1.503226567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:26.503063541 +0000 UTC m=+2822.496227693" watchObservedRunningTime="2026-07-16 22:34:26.503226567 +0000 UTC m=+2822.496390718"
Jul 16 22:34:27.489875 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:27.489835    2555 generic.go:356] "Generic (PLEG): container finished" podID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerID="edf6692017bd2ad4e3635ab5b378bb22e59f4e5a11eac2832e6dab17e040b7a6" exitCode=6
Jul 16 22:34:27.490339 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:27.489926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" event={"ID":"c43ca8bc-ae65-4527-b803-7f99a1f7a431","Type":"ContainerDied","Data":"edf6692017bd2ad4e3635ab5b378bb22e59f4e5a11eac2832e6dab17e040b7a6"}
Jul 16 22:34:27.490339 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:27.490135    2555 scope.go:117] "RemoveContainer" containerID="edf6692017bd2ad4e3635ab5b378bb22e59f4e5a11eac2832e6dab17e040b7a6"
Jul 16 22:34:27.491506 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:27.491486    2555 generic.go:356] "Generic (PLEG): container finished" podID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerID="21826bdde78b1f56e94684d19b00b34ab5e53dd3baf9fc90c075a10540c357c7" exitCode=6
Jul 16 22:34:27.491608 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:27.491569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" event={"ID":"966175f6-c81e-484c-ad10-0bb3a0b7a595","Type":"ContainerDied","Data":"21826bdde78b1f56e94684d19b00b34ab5e53dd3baf9fc90c075a10540c357c7"}
Jul 16 22:34:27.491709 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:27.491696    2555 scope.go:117] "RemoveContainer" containerID="21826bdde78b1f56e94684d19b00b34ab5e53dd3baf9fc90c075a10540c357c7"
Jul 16 22:34:28.497040 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.496939    2555 generic.go:356] "Generic (PLEG): container finished" podID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerID="13e0e072d343179c051abab2b4cade043f74566a64ce9a811ca5071fe0548678" exitCode=6
Jul 16 22:34:28.497040 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.497021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" event={"ID":"c43ca8bc-ae65-4527-b803-7f99a1f7a431","Type":"ContainerDied","Data":"13e0e072d343179c051abab2b4cade043f74566a64ce9a811ca5071fe0548678"}
Jul 16 22:34:28.497559 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.497086    2555 scope.go:117] "RemoveContainer" containerID="edf6692017bd2ad4e3635ab5b378bb22e59f4e5a11eac2832e6dab17e040b7a6"
Jul 16 22:34:28.497559 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.497281    2555 scope.go:117] "RemoveContainer" containerID="13e0e072d343179c051abab2b4cade043f74566a64ce9a811ca5071fe0548678"
Jul 16 22:34:28.497559 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:28.497496    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s_odh-ai-gateway-infra(c43ca8bc-ae65-4527-b803-7f99a1f7a431)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431"
Jul 16 22:34:28.498691 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.498672    2555 generic.go:356] "Generic (PLEG): container finished" podID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerID="183ac8bf538d6bbdf8bcbba9e4b832f9ef2558e33ec658808cca4abf6b1d2fc4" exitCode=6
Jul 16 22:34:28.498775 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.498733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" event={"ID":"966175f6-c81e-484c-ad10-0bb3a0b7a595","Type":"ContainerDied","Data":"183ac8bf538d6bbdf8bcbba9e4b832f9ef2558e33ec658808cca4abf6b1d2fc4"}
Jul 16 22:34:28.498911 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.498898    2555 scope.go:117] "RemoveContainer" containerID="183ac8bf538d6bbdf8bcbba9e4b832f9ef2558e33ec658808cca4abf6b1d2fc4"
Jul 16 22:34:28.499070 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:28.499053    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7_odh-ai-gateway-infra(966175f6-c81e-484c-ad10-0bb3a0b7a595)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595"
Jul 16 22:34:28.507356 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.507340    2555 scope.go:117] "RemoveContainer" containerID="21826bdde78b1f56e94684d19b00b34ab5e53dd3baf9fc90c075a10540c357c7"
Jul 16 22:34:28.572199 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.572165    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"]
Jul 16 22:34:28.576410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.576395    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"
Jul 16 22:34:28.599338 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.599307    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"]
Jul 16 22:34:28.625580 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.625552    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4d33a-a928-41e1-b58c-71cf73867ee4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8\" (UID: \"b0d4d33a-a928-41e1-b58c-71cf73867ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"
Jul 16 22:34:28.726736 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.726698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4d33a-a928-41e1-b58c-71cf73867ee4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8\" (UID: \"b0d4d33a-a928-41e1-b58c-71cf73867ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"
Jul 16 22:34:28.727400 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.727378    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4d33a-a928-41e1-b58c-71cf73867ee4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8\" (UID: \"b0d4d33a-a928-41e1-b58c-71cf73867ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"
Jul 16 22:34:28.886186 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:28.886147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"
Jul 16 22:34:29.022917 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:29.022891    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"]
Jul 16 22:34:29.024755 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:34:29.024727    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d4d33a_a928_41e1_b58c_71cf73867ee4.slice/crio-33784bb6c99f7704170de899a8ca95043ac0b81221fc6df08f43a0d39756f995 WatchSource:0}: Error finding container 33784bb6c99f7704170de899a8ca95043ac0b81221fc6df08f43a0d39756f995: Status 404 returned error can't find the container with id 33784bb6c99f7704170de899a8ca95043ac0b81221fc6df08f43a0d39756f995
Jul 16 22:34:29.504910 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:29.504878    2555 scope.go:117] "RemoveContainer" containerID="13e0e072d343179c051abab2b4cade043f74566a64ce9a811ca5071fe0548678"
Jul 16 22:34:29.505404 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:29.505154    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s_odh-ai-gateway-infra(c43ca8bc-ae65-4527-b803-7f99a1f7a431)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431"
Jul 16 22:34:29.506545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:29.506524    2555 scope.go:117] "RemoveContainer" containerID="183ac8bf538d6bbdf8bcbba9e4b832f9ef2558e33ec658808cca4abf6b1d2fc4"
Jul 16 22:34:29.506722 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:29.506693    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7_odh-ai-gateway-infra(966175f6-c81e-484c-ad10-0bb3a0b7a595)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595"
Jul 16 22:34:29.507815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:29.507790    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" event={"ID":"b0d4d33a-a928-41e1-b58c-71cf73867ee4","Type":"ContainerStarted","Data":"885ac279f8297a54ee15464a034f4e43b7ea37f478d6177e3b0821c6ac1c086b"}
Jul 16 22:34:29.507920 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:29.507834    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" event={"ID":"b0d4d33a-a928-41e1-b58c-71cf73867ee4","Type":"ContainerStarted","Data":"33784bb6c99f7704170de899a8ca95043ac0b81221fc6df08f43a0d39756f995"}
Jul 16 22:34:29.558214 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:29.558108    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" podStartSLOduration=1.558092365 podStartE2EDuration="1.558092365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:29.556311088 +0000 UTC m=+2825.549475240" watchObservedRunningTime="2026-07-16 22:34:29.558092365 +0000 UTC m=+2825.551256569"
Jul 16 22:34:30.512865 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:30.512830    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerID="885ac279f8297a54ee15464a034f4e43b7ea37f478d6177e3b0821c6ac1c086b" exitCode=6
Jul 16 22:34:30.513261 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:30.512920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" event={"ID":"b0d4d33a-a928-41e1-b58c-71cf73867ee4","Type":"ContainerDied","Data":"885ac279f8297a54ee15464a034f4e43b7ea37f478d6177e3b0821c6ac1c086b"}
Jul 16 22:34:30.513261 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:30.513120    2555 scope.go:117] "RemoveContainer" containerID="885ac279f8297a54ee15464a034f4e43b7ea37f478d6177e3b0821c6ac1c086b"
Jul 16 22:34:31.520051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:31.519939    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerID="a2e4e1bb67bbd0d48c7c2c260050213c979746f51033d430421a0a01546ec6ea" exitCode=6
Jul 16 22:34:31.520051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:31.519990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" event={"ID":"b0d4d33a-a928-41e1-b58c-71cf73867ee4","Type":"ContainerDied","Data":"a2e4e1bb67bbd0d48c7c2c260050213c979746f51033d430421a0a01546ec6ea"}
Jul 16 22:34:31.520051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:31.520053    2555 scope.go:117] "RemoveContainer" containerID="885ac279f8297a54ee15464a034f4e43b7ea37f478d6177e3b0821c6ac1c086b"
Jul 16 22:34:31.520608 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:31.520252    2555 scope.go:117] "RemoveContainer" containerID="a2e4e1bb67bbd0d48c7c2c260050213c979746f51033d430421a0a01546ec6ea"
Jul 16 22:34:31.520608 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:31.520482    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8_odh-ai-gateway-infra(b0d4d33a-a928-41e1-b58c-71cf73867ee4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4"
Jul 16 22:34:32.525882 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:32.525860    2555 scope.go:117] "RemoveContainer" containerID="a2e4e1bb67bbd0d48c7c2c260050213c979746f51033d430421a0a01546ec6ea"
Jul 16 22:34:32.526280 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:32.526073    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8_odh-ai-gateway-infra(b0d4d33a-a928-41e1-b58c-71cf73867ee4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4"
Jul 16 22:34:40.489917 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:40.489880    2555 scope.go:117] "RemoveContainer" containerID="183ac8bf538d6bbdf8bcbba9e4b832f9ef2558e33ec658808cca4abf6b1d2fc4"
Jul 16 22:34:41.566676 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:41.566593    2555 generic.go:356] "Generic (PLEG): container finished" podID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerID="acfa4f9a2b13427d6202696fd2e8dc14178728db7a22e49717632f494c46ee7f" exitCode=6
Jul 16 22:34:41.567135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:41.566668    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" event={"ID":"966175f6-c81e-484c-ad10-0bb3a0b7a595","Type":"ContainerDied","Data":"acfa4f9a2b13427d6202696fd2e8dc14178728db7a22e49717632f494c46ee7f"}
Jul 16 22:34:41.567135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:41.566730    2555 scope.go:117] "RemoveContainer" containerID="183ac8bf538d6bbdf8bcbba9e4b832f9ef2558e33ec658808cca4abf6b1d2fc4"
Jul 16 22:34:41.567135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:41.567012    2555 scope.go:117] "RemoveContainer" containerID="acfa4f9a2b13427d6202696fd2e8dc14178728db7a22e49717632f494c46ee7f"
Jul 16 22:34:41.567277 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:41.567257    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7_odh-ai-gateway-infra(966175f6-c81e-484c-ad10-0bb3a0b7a595)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595"
Jul 16 22:34:42.594574 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:42.594535    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"]
Jul 16 22:34:42.740767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:42.740741    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"
Jul 16 22:34:42.853173 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:42.853081    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/966175f6-c81e-484c-ad10-0bb3a0b7a595-maas-api-service-ca\") pod \"966175f6-c81e-484c-ad10-0bb3a0b7a595\" (UID: \"966175f6-c81e-484c-ad10-0bb3a0b7a595\") "
Jul 16 22:34:42.853489 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:42.853467    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966175f6-c81e-484c-ad10-0bb3a0b7a595-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "966175f6-c81e-484c-ad10-0bb3a0b7a595" (UID: "966175f6-c81e-484c-ad10-0bb3a0b7a595"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:42.954482 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:42.954439    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/966175f6-c81e-484c-ad10-0bb3a0b7a595-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:34:43.489898 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:43.489864    2555 scope.go:117] "RemoveContainer" containerID="13e0e072d343179c051abab2b4cade043f74566a64ce9a811ca5071fe0548678"
Jul 16 22:34:43.577607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:43.577582    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"
Jul 16 22:34:43.577607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:43.577588    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7" event={"ID":"966175f6-c81e-484c-ad10-0bb3a0b7a595","Type":"ContainerDied","Data":"b14e2940bb4da2d2652df1a7cece6bae09c628217a6d05c5a4ffa7ad1762737d"}
Jul 16 22:34:43.577826 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:43.577650    2555 scope.go:117] "RemoveContainer" containerID="acfa4f9a2b13427d6202696fd2e8dc14178728db7a22e49717632f494c46ee7f"
Jul 16 22:34:43.600611 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:43.600576    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"]
Jul 16 22:34:43.605062 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:43.605036    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9tmw7"]
Jul 16 22:34:44.494770 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:44.494693    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" path="/var/lib/kubelet/pods/966175f6-c81e-484c-ad10-0bb3a0b7a595/volumes"
Jul 16 22:34:44.583673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:44.583640    2555 generic.go:356] "Generic (PLEG): container finished" podID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerID="181e6c9d3b0d9ea089fa842be24d4e622e9140d2fa2a0af801d3251e4b0c5539" exitCode=6
Jul 16 22:34:44.583840 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:44.583708    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" event={"ID":"c43ca8bc-ae65-4527-b803-7f99a1f7a431","Type":"ContainerDied","Data":"181e6c9d3b0d9ea089fa842be24d4e622e9140d2fa2a0af801d3251e4b0c5539"}
Jul 16 22:34:44.583840 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:44.583769    2555 scope.go:117] "RemoveContainer" containerID="13e0e072d343179c051abab2b4cade043f74566a64ce9a811ca5071fe0548678"
Jul 16 22:34:44.584128 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:44.584110    2555 scope.go:117] "RemoveContainer" containerID="181e6c9d3b0d9ea089fa842be24d4e622e9140d2fa2a0af801d3251e4b0c5539"
Jul 16 22:34:44.584384 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:44.584355    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s_odh-ai-gateway-infra(c43ca8bc-ae65-4527-b803-7f99a1f7a431)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431"
Jul 16 22:34:45.489582 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:45.489552    2555 scope.go:117] "RemoveContainer" containerID="a2e4e1bb67bbd0d48c7c2c260050213c979746f51033d430421a0a01546ec6ea"
Jul 16 22:34:45.610873 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:45.610840    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"]
Jul 16 22:34:45.731019 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:45.730992    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"
Jul 16 22:34:45.878883 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:45.878845    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ca8bc-ae65-4527-b803-7f99a1f7a431-maas-api-service-ca\") pod \"c43ca8bc-ae65-4527-b803-7f99a1f7a431\" (UID: \"c43ca8bc-ae65-4527-b803-7f99a1f7a431\") "
Jul 16 22:34:45.879260 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:45.879235    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43ca8bc-ae65-4527-b803-7f99a1f7a431-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c43ca8bc-ae65-4527-b803-7f99a1f7a431" (UID: "c43ca8bc-ae65-4527-b803-7f99a1f7a431"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:45.980416 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:45.980385    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ca8bc-ae65-4527-b803-7f99a1f7a431-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:34:46.596719 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.596687    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerID="870fd0d90b7f25ce3d40d98a9fef0568359796f5fa423d99de3ad0e75baa0895" exitCode=6
Jul 16 22:34:46.597194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.596759    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" event={"ID":"b0d4d33a-a928-41e1-b58c-71cf73867ee4","Type":"ContainerDied","Data":"870fd0d90b7f25ce3d40d98a9fef0568359796f5fa423d99de3ad0e75baa0895"}
Jul 16 22:34:46.597194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.596823    2555 scope.go:117] "RemoveContainer" containerID="a2e4e1bb67bbd0d48c7c2c260050213c979746f51033d430421a0a01546ec6ea"
Jul 16 22:34:46.597194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.597122    2555 scope.go:117] "RemoveContainer" containerID="870fd0d90b7f25ce3d40d98a9fef0568359796f5fa423d99de3ad0e75baa0895"
Jul 16 22:34:46.597371 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:46.597348    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8_odh-ai-gateway-infra(b0d4d33a-a928-41e1-b58c-71cf73867ee4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4"
Jul 16 22:34:46.598356 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.598332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s" event={"ID":"c43ca8bc-ae65-4527-b803-7f99a1f7a431","Type":"ContainerDied","Data":"c02842a19261ca0f5c3dd7f1db8d8cca9682bb5388bfcc2c0d62fa023865ea33"}
Jul 16 22:34:46.598439 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.598368    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"
Jul 16 22:34:46.607410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.607398    2555 scope.go:117] "RemoveContainer" containerID="181e6c9d3b0d9ea089fa842be24d4e622e9140d2fa2a0af801d3251e4b0c5539"
Jul 16 22:34:46.631231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.631209    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"]
Jul 16 22:34:46.636898 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:46.636872    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5xj5s"]
Jul 16 22:34:47.624727 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:47.624698    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"]
Jul 16 22:34:47.760086 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:47.760059    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"
Jul 16 22:34:47.897004 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:47.896897    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4d33a-a928-41e1-b58c-71cf73867ee4-maas-api-service-ca\") pod \"b0d4d33a-a928-41e1-b58c-71cf73867ee4\" (UID: \"b0d4d33a-a928-41e1-b58c-71cf73867ee4\") "
Jul 16 22:34:47.897286 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:47.897263    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d4d33a-a928-41e1-b58c-71cf73867ee4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0d4d33a-a928-41e1-b58c-71cf73867ee4" (UID: "b0d4d33a-a928-41e1-b58c-71cf73867ee4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:47.997554 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:47.997521    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d4d33a-a928-41e1-b58c-71cf73867ee4-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:34:48.494990 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:48.494922    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" path="/var/lib/kubelet/pods/c43ca8bc-ae65-4527-b803-7f99a1f7a431/volumes"
Jul 16 22:34:48.612366 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:48.612332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8" event={"ID":"b0d4d33a-a928-41e1-b58c-71cf73867ee4","Type":"ContainerDied","Data":"33784bb6c99f7704170de899a8ca95043ac0b81221fc6df08f43a0d39756f995"}
Jul 16 22:34:48.612568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:48.612403    2555 scope.go:117] "RemoveContainer" containerID="870fd0d90b7f25ce3d40d98a9fef0568359796f5fa423d99de3ad0e75baa0895"
Jul 16 22:34:48.612568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:48.612343    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"
Jul 16 22:34:48.631712 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:48.631691    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"]
Jul 16 22:34:48.650475 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:48.650440    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-9x6f8"]
Jul 16 22:34:50.494525 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:50.494490    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" path="/var/lib/kubelet/pods/b0d4d33a-a928-41e1-b58c-71cf73867ee4/volumes"
Jul 16 22:34:57.222932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.222892    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"]
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223221    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223236    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223247    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223252    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223260    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223268    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223275    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223281    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223286    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223291    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223310    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223316    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223331    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223337    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223342    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223396    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223405    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223410    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223415    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223422    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:34:57.223430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.223432    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="966175f6-c81e-484c-ad10-0bb3a0b7a595" containerName="revoke-keys"
Jul 16 22:34:57.227564 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.227544    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"
Jul 16 22:34:57.230435 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.230408    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:34:57.230863 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.230849    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:34:57.236090 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.236066    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"]
Jul 16 22:34:57.379547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.379481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54c32e8-ff4b-4354-bfae-8434d2176bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s\" (UID: \"f54c32e8-ff4b-4354-bfae-8434d2176bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"
Jul 16 22:34:57.480063 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.479946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54c32e8-ff4b-4354-bfae-8434d2176bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s\" (UID: \"f54c32e8-ff4b-4354-bfae-8434d2176bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"
Jul 16 22:34:57.480577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.480556    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54c32e8-ff4b-4354-bfae-8434d2176bdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s\" (UID: \"f54c32e8-ff4b-4354-bfae-8434d2176bdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"
Jul 16 22:34:57.538378 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.538344    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"
Jul 16 22:34:57.674666 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:57.674635    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"]
Jul 16 22:34:57.675568 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:34:57.675544    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54c32e8_ff4b_4354_bfae_8434d2176bdc.slice/crio-752c68a33955e8d94abe3c977510b45272779098d9fdcae1e2acca85637c7076 WatchSource:0}: Error finding container 752c68a33955e8d94abe3c977510b45272779098d9fdcae1e2acca85637c7076: Status 404 returned error can't find the container with id 752c68a33955e8d94abe3c977510b45272779098d9fdcae1e2acca85637c7076
Jul 16 22:34:58.656445 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:58.656416    2555 generic.go:356] "Generic (PLEG): container finished" podID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerID="b587c62f2abe1a5ee84e59a77edfeea2171759d408a73c0de616ad15aaba8ef1" exitCode=6
Jul 16 22:34:58.656821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:58.656503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" event={"ID":"f54c32e8-ff4b-4354-bfae-8434d2176bdc","Type":"ContainerDied","Data":"b587c62f2abe1a5ee84e59a77edfeea2171759d408a73c0de616ad15aaba8ef1"}
Jul 16 22:34:58.656821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:58.656555    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" event={"ID":"f54c32e8-ff4b-4354-bfae-8434d2176bdc","Type":"ContainerStarted","Data":"752c68a33955e8d94abe3c977510b45272779098d9fdcae1e2acca85637c7076"}
Jul 16 22:34:58.656821 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:58.656646    2555 scope.go:117] "RemoveContainer" containerID="b587c62f2abe1a5ee84e59a77edfeea2171759d408a73c0de616ad15aaba8ef1"
Jul 16 22:34:59.662146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:59.662113    2555 generic.go:356] "Generic (PLEG): container finished" podID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerID="bed5d873ba7fb7e62b15f48ba50c69c72021cdc6fb1694e085dc046171a67532" exitCode=6
Jul 16 22:34:59.662545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:59.662176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" event={"ID":"f54c32e8-ff4b-4354-bfae-8434d2176bdc","Type":"ContainerDied","Data":"bed5d873ba7fb7e62b15f48ba50c69c72021cdc6fb1694e085dc046171a67532"}
Jul 16 22:34:59.662545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:59.662247    2555 scope.go:117] "RemoveContainer" containerID="b587c62f2abe1a5ee84e59a77edfeea2171759d408a73c0de616ad15aaba8ef1"
Jul 16 22:34:59.662545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:34:59.662497    2555 scope.go:117] "RemoveContainer" containerID="bed5d873ba7fb7e62b15f48ba50c69c72021cdc6fb1694e085dc046171a67532"
Jul 16 22:34:59.662732 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:34:59.662712    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s_odh-ai-gateway-infra(f54c32e8-ff4b-4354-bfae-8434d2176bdc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc"
Jul 16 22:35:00.668823 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:00.668796    2555 scope.go:117] "RemoveContainer" containerID="bed5d873ba7fb7e62b15f48ba50c69c72021cdc6fb1694e085dc046171a67532"
Jul 16 22:35:00.669212 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:00.669065    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s_odh-ai-gateway-infra(f54c32e8-ff4b-4354-bfae-8434d2176bdc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc"
Jul 16 22:35:12.489577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:12.489543    2555 scope.go:117] "RemoveContainer" containerID="bed5d873ba7fb7e62b15f48ba50c69c72021cdc6fb1694e085dc046171a67532"
Jul 16 22:35:13.727296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:13.727258    2555 generic.go:356] "Generic (PLEG): container finished" podID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerID="b7cad62ca67363e4c54416bb4ed4e08a384717dd4b67e8822df9ca9fc26dd885" exitCode=6
Jul 16 22:35:13.727777 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:13.727331    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" event={"ID":"f54c32e8-ff4b-4354-bfae-8434d2176bdc","Type":"ContainerDied","Data":"b7cad62ca67363e4c54416bb4ed4e08a384717dd4b67e8822df9ca9fc26dd885"}
Jul 16 22:35:13.727777 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:13.727395    2555 scope.go:117] "RemoveContainer" containerID="bed5d873ba7fb7e62b15f48ba50c69c72021cdc6fb1694e085dc046171a67532"
Jul 16 22:35:13.727777 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:13.727725    2555 scope.go:117] "RemoveContainer" containerID="b7cad62ca67363e4c54416bb4ed4e08a384717dd4b67e8822df9ca9fc26dd885"
Jul 16 22:35:13.727994 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:13.727974    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s_odh-ai-gateway-infra(f54c32e8-ff4b-4354-bfae-8434d2176bdc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc"
Jul 16 22:35:14.775707 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:14.775674    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"]
Jul 16 22:35:14.923471 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:14.923445    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"
Jul 16 22:35:15.015714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.015684    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54c32e8-ff4b-4354-bfae-8434d2176bdc-maas-api-service-ca\") pod \"f54c32e8-ff4b-4354-bfae-8434d2176bdc\" (UID: \"f54c32e8-ff4b-4354-bfae-8434d2176bdc\") "
Jul 16 22:35:15.016069 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.016045    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c32e8-ff4b-4354-bfae-8434d2176bdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f54c32e8-ff4b-4354-bfae-8434d2176bdc" (UID: "f54c32e8-ff4b-4354-bfae-8434d2176bdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:15.116212 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.116182    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54c32e8-ff4b-4354-bfae-8434d2176bdc-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:35:15.738670 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.738643    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"
Jul 16 22:35:15.738839 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.738639    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s" event={"ID":"f54c32e8-ff4b-4354-bfae-8434d2176bdc","Type":"ContainerDied","Data":"752c68a33955e8d94abe3c977510b45272779098d9fdcae1e2acca85637c7076"}
Jul 16 22:35:15.738839 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.738784    2555 scope.go:117] "RemoveContainer" containerID="b7cad62ca67363e4c54416bb4ed4e08a384717dd4b67e8822df9ca9fc26dd885"
Jul 16 22:35:15.759102 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.759072    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"]
Jul 16 22:35:15.763077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:15.763057    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-pzv4s"]
Jul 16 22:35:16.115331 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115300    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"]
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115601    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115615    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115628    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115634    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115644    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115649    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115698    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115706    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:16.115710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115714    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d4d33a-a928-41e1-b58c-71cf73867ee4" containerName="revoke-keys"
Jul 16 22:35:16.116024 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.115721    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43ca8bc-ae65-4527-b803-7f99a1f7a431" containerName="revoke-keys"
Jul 16 22:35:16.119778 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.119762    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"
Jul 16 22:35:16.122650 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.122626    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:35:16.122791 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.122735    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:35:16.124654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.124631    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21fb1b-d8db-4c02-b153-22cdb89c3251-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r\" (UID: \"8f21fb1b-d8db-4c02-b153-22cdb89c3251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"
Jul 16 22:35:16.126484 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.126463    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"]
Jul 16 22:35:16.225136 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.225088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21fb1b-d8db-4c02-b153-22cdb89c3251-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r\" (UID: \"8f21fb1b-d8db-4c02-b153-22cdb89c3251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"
Jul 16 22:35:16.225681 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.225656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21fb1b-d8db-4c02-b153-22cdb89c3251-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r\" (UID: \"8f21fb1b-d8db-4c02-b153-22cdb89c3251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"
Jul 16 22:35:16.431109 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.431024    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"
Jul 16 22:35:16.495801 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.495769    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" path="/var/lib/kubelet/pods/f54c32e8-ff4b-4354-bfae-8434d2176bdc/volumes"
Jul 16 22:35:16.565285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.565261    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"]
Jul 16 22:35:16.567290 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:35:16.567257    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f21fb1b_d8db_4c02_b153_22cdb89c3251.slice/crio-58eb8ec74fa369282f9e2c6cec6337263e36ecf010c03aa67a3672a3fe6f059e WatchSource:0}: Error finding container 58eb8ec74fa369282f9e2c6cec6337263e36ecf010c03aa67a3672a3fe6f059e: Status 404 returned error can't find the container with id 58eb8ec74fa369282f9e2c6cec6337263e36ecf010c03aa67a3672a3fe6f059e
Jul 16 22:35:16.744270 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:16.744173    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" event={"ID":"8f21fb1b-d8db-4c02-b153-22cdb89c3251","Type":"ContainerStarted","Data":"58eb8ec74fa369282f9e2c6cec6337263e36ecf010c03aa67a3672a3fe6f059e"}
Jul 16 22:35:17.752387 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:17.752298    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerID="fafb29c224566fcc0812d220dffacc8dc7a9c2bd2b9e10f00e5a78b5006d4c73" exitCode=6
Jul 16 22:35:17.752740 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:17.752377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" event={"ID":"8f21fb1b-d8db-4c02-b153-22cdb89c3251","Type":"ContainerDied","Data":"fafb29c224566fcc0812d220dffacc8dc7a9c2bd2b9e10f00e5a78b5006d4c73"}
Jul 16 22:35:17.752740 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:17.752602    2555 scope.go:117] "RemoveContainer" containerID="fafb29c224566fcc0812d220dffacc8dc7a9c2bd2b9e10f00e5a78b5006d4c73"
Jul 16 22:35:18.759009 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:18.758949    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerID="acaed1d9eaa3a77993c0a5d44217726ae9dd06f1661345040b37d0e8abb158ca" exitCode=6
Jul 16 22:35:18.759009 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:18.758988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" event={"ID":"8f21fb1b-d8db-4c02-b153-22cdb89c3251","Type":"ContainerDied","Data":"acaed1d9eaa3a77993c0a5d44217726ae9dd06f1661345040b37d0e8abb158ca"}
Jul 16 22:35:18.759536 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:18.759064    2555 scope.go:117] "RemoveContainer" containerID="fafb29c224566fcc0812d220dffacc8dc7a9c2bd2b9e10f00e5a78b5006d4c73"
Jul 16 22:35:18.759536 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:18.759215    2555 scope.go:117] "RemoveContainer" containerID="acaed1d9eaa3a77993c0a5d44217726ae9dd06f1661345040b37d0e8abb158ca"
Jul 16 22:35:18.759536 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:18.759444    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r_odh-ai-gateway-infra(8f21fb1b-d8db-4c02-b153-22cdb89c3251)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251"
Jul 16 22:35:19.765252 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:19.765226    2555 scope.go:117] "RemoveContainer" containerID="acaed1d9eaa3a77993c0a5d44217726ae9dd06f1661345040b37d0e8abb158ca"
Jul 16 22:35:19.765633 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:19.765402    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r_odh-ai-gateway-infra(8f21fb1b-d8db-4c02-b153-22cdb89c3251)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251"
Jul 16 22:35:25.539820 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.539781    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"]
Jul 16 22:35:25.540258 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.540242    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:25.540299 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.540262    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:25.540349 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.540338    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c32e8-ff4b-4354-bfae-8434d2176bdc" containerName="revoke-keys"
Jul 16 22:35:25.545469 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.545450    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"
Jul 16 22:35:25.556672 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.556646    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"]
Jul 16 22:35:25.609149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.609115    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b1a8d-cc42-40b5-a82e-ad1161d94de5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5jctf\" (UID: \"291b1a8d-cc42-40b5-a82e-ad1161d94de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"
Jul 16 22:35:25.710069 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.710036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b1a8d-cc42-40b5-a82e-ad1161d94de5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5jctf\" (UID: \"291b1a8d-cc42-40b5-a82e-ad1161d94de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"
Jul 16 22:35:25.710607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.710587    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b1a8d-cc42-40b5-a82e-ad1161d94de5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-5jctf\" (UID: \"291b1a8d-cc42-40b5-a82e-ad1161d94de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"
Jul 16 22:35:25.855279 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.855248    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"
Jul 16 22:35:25.996919 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:35:25.996886    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291b1a8d_cc42_40b5_a82e_ad1161d94de5.slice/crio-68922e4d85f5b2ed98c155cfaed869225493e9983c5da51948c16d1830c16fc6 WatchSource:0}: Error finding container 68922e4d85f5b2ed98c155cfaed869225493e9983c5da51948c16d1830c16fc6: Status 404 returned error can't find the container with id 68922e4d85f5b2ed98c155cfaed869225493e9983c5da51948c16d1830c16fc6
Jul 16 22:35:25.998276 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:25.998254    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"]
Jul 16 22:35:26.795146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:26.795063    2555 generic.go:356] "Generic (PLEG): container finished" podID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerID="00ca148a44a401ccbc5ac78b5ba009fa6e2b216cce393489c76245b57fe5a9ce" exitCode=6
Jul 16 22:35:26.795146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:26.795104    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" event={"ID":"291b1a8d-cc42-40b5-a82e-ad1161d94de5","Type":"ContainerDied","Data":"00ca148a44a401ccbc5ac78b5ba009fa6e2b216cce393489c76245b57fe5a9ce"}
Jul 16 22:35:26.795146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:26.795139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" event={"ID":"291b1a8d-cc42-40b5-a82e-ad1161d94de5","Type":"ContainerStarted","Data":"68922e4d85f5b2ed98c155cfaed869225493e9983c5da51948c16d1830c16fc6"}
Jul 16 22:35:26.795564 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:26.795322    2555 scope.go:117] "RemoveContainer" containerID="00ca148a44a401ccbc5ac78b5ba009fa6e2b216cce393489c76245b57fe5a9ce"
Jul 16 22:35:27.801529 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:27.801446    2555 generic.go:356] "Generic (PLEG): container finished" podID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerID="3157e711d07e0168afb5191f9c3df9e67722b91d50434781859db2e9e32cf666" exitCode=6
Jul 16 22:35:27.801919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:27.801533    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" event={"ID":"291b1a8d-cc42-40b5-a82e-ad1161d94de5","Type":"ContainerDied","Data":"3157e711d07e0168afb5191f9c3df9e67722b91d50434781859db2e9e32cf666"}
Jul 16 22:35:27.801919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:27.801595    2555 scope.go:117] "RemoveContainer" containerID="00ca148a44a401ccbc5ac78b5ba009fa6e2b216cce393489c76245b57fe5a9ce"
Jul 16 22:35:27.801919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:27.801784    2555 scope.go:117] "RemoveContainer" containerID="3157e711d07e0168afb5191f9c3df9e67722b91d50434781859db2e9e32cf666"
Jul 16 22:35:27.802112 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:27.802054    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5jctf_odh-ai-gateway-infra(291b1a8d-cc42-40b5-a82e-ad1161d94de5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5"
Jul 16 22:35:28.566907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.566873    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"]
Jul 16 22:35:28.570817 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.570800    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"
Jul 16 22:35:28.581733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.581693    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"]
Jul 16 22:35:28.639097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.639059    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d0b87-6d79-460e-be02-b41aaba37f09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk\" (UID: \"5c6d0b87-6d79-460e-be02-b41aaba37f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"
Jul 16 22:35:28.740363 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.740327    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d0b87-6d79-460e-be02-b41aaba37f09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk\" (UID: \"5c6d0b87-6d79-460e-be02-b41aaba37f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"
Jul 16 22:35:28.740981 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.740935    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d0b87-6d79-460e-be02-b41aaba37f09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk\" (UID: \"5c6d0b87-6d79-460e-be02-b41aaba37f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"
Jul 16 22:35:28.807914 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.807887    2555 scope.go:117] "RemoveContainer" containerID="3157e711d07e0168afb5191f9c3df9e67722b91d50434781859db2e9e32cf666"
Jul 16 22:35:28.808293 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:28.808103    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5jctf_odh-ai-gateway-infra(291b1a8d-cc42-40b5-a82e-ad1161d94de5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5"
Jul 16 22:35:28.884285 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:28.884250    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"
Jul 16 22:35:29.013969 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:29.013934    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"]
Jul 16 22:35:29.016423 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:35:29.016389    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6d0b87_6d79_460e_be02_b41aaba37f09.slice/crio-8721d4281a335125ad459770e1b8104a0e2a69c053889fededc0b72a35189cd7 WatchSource:0}: Error finding container 8721d4281a335125ad459770e1b8104a0e2a69c053889fededc0b72a35189cd7: Status 404 returned error can't find the container with id 8721d4281a335125ad459770e1b8104a0e2a69c053889fededc0b72a35189cd7
Jul 16 22:35:29.814289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:29.814259    2555 generic.go:356] "Generic (PLEG): container finished" podID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerID="caca9c803afe28d5899057b6b996645836c7aa2d0fb919378e696a440689356e" exitCode=6
Jul 16 22:35:29.814637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:29.814311    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" event={"ID":"5c6d0b87-6d79-460e-be02-b41aaba37f09","Type":"ContainerDied","Data":"caca9c803afe28d5899057b6b996645836c7aa2d0fb919378e696a440689356e"}
Jul 16 22:35:29.814637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:29.814361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" event={"ID":"5c6d0b87-6d79-460e-be02-b41aaba37f09","Type":"ContainerStarted","Data":"8721d4281a335125ad459770e1b8104a0e2a69c053889fededc0b72a35189cd7"}
Jul 16 22:35:29.814637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:29.814550    2555 scope.go:117] "RemoveContainer" containerID="caca9c803afe28d5899057b6b996645836c7aa2d0fb919378e696a440689356e"
Jul 16 22:35:30.820365 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:30.820271    2555 generic.go:356] "Generic (PLEG): container finished" podID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerID="e9f55265a4fe062d77337d71d1deaf2b593b47e55739c06ccaba696695d6bf26" exitCode=6
Jul 16 22:35:30.820365 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:30.820337    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" event={"ID":"5c6d0b87-6d79-460e-be02-b41aaba37f09","Type":"ContainerDied","Data":"e9f55265a4fe062d77337d71d1deaf2b593b47e55739c06ccaba696695d6bf26"}
Jul 16 22:35:30.820831 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:30.820397    2555 scope.go:117] "RemoveContainer" containerID="caca9c803afe28d5899057b6b996645836c7aa2d0fb919378e696a440689356e"
Jul 16 22:35:30.820831 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:30.820568    2555 scope.go:117] "RemoveContainer" containerID="e9f55265a4fe062d77337d71d1deaf2b593b47e55739c06ccaba696695d6bf26"
Jul 16 22:35:30.820831 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:30.820767    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk_odh-ai-gateway-infra(5c6d0b87-6d79-460e-be02-b41aaba37f09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09"
Jul 16 22:35:31.825459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:31.825436    2555 scope.go:117] "RemoveContainer" containerID="e9f55265a4fe062d77337d71d1deaf2b593b47e55739c06ccaba696695d6bf26"
Jul 16 22:35:31.825850 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:31.825625    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk_odh-ai-gateway-infra(5c6d0b87-6d79-460e-be02-b41aaba37f09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09"
Jul 16 22:35:32.494786 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:32.494760    2555 scope.go:117] "RemoveContainer" containerID="acaed1d9eaa3a77993c0a5d44217726ae9dd06f1661345040b37d0e8abb158ca"
Jul 16 22:35:32.832593 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:32.832561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" event={"ID":"8f21fb1b-d8db-4c02-b153-22cdb89c3251","Type":"ContainerStarted","Data":"d0dbe5207d7b6ddb6dff3bb3d257c1f7d51f8fd7afaeb8ff5a5b55d58a96f5e5"}
Jul 16 22:35:32.862930 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:32.862882    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" podStartSLOduration=16.862865662 podStartE2EDuration="16.862865662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:32.860450408 +0000 UTC m=+2888.853614561" watchObservedRunningTime="2026-07-16 22:35:32.862865662 +0000 UTC m=+2888.856029813"
Jul 16 22:35:33.840051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:33.840019    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerID="d0dbe5207d7b6ddb6dff3bb3d257c1f7d51f8fd7afaeb8ff5a5b55d58a96f5e5" exitCode=6
Jul 16 22:35:33.840503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:33.840057    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" event={"ID":"8f21fb1b-d8db-4c02-b153-22cdb89c3251","Type":"ContainerDied","Data":"d0dbe5207d7b6ddb6dff3bb3d257c1f7d51f8fd7afaeb8ff5a5b55d58a96f5e5"}
Jul 16 22:35:33.840503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:33.840132    2555 scope.go:117] "RemoveContainer" containerID="acaed1d9eaa3a77993c0a5d44217726ae9dd06f1661345040b37d0e8abb158ca"
Jul 16 22:35:33.840503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:33.840407    2555 scope.go:117] "RemoveContainer" containerID="d0dbe5207d7b6ddb6dff3bb3d257c1f7d51f8fd7afaeb8ff5a5b55d58a96f5e5"
Jul 16 22:35:33.840665 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:33.840643    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r_odh-ai-gateway-infra(8f21fb1b-d8db-4c02-b153-22cdb89c3251)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251"
Jul 16 22:35:33.874686 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:33.874657    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"]
Jul 16 22:35:34.978353 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:34.978330    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"
Jul 16 22:35:35.096033 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.095997    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21fb1b-d8db-4c02-b153-22cdb89c3251-maas-api-service-ca\") pod \"8f21fb1b-d8db-4c02-b153-22cdb89c3251\" (UID: \"8f21fb1b-d8db-4c02-b153-22cdb89c3251\") "
Jul 16 22:35:35.096395 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.096344    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f21fb1b-d8db-4c02-b153-22cdb89c3251-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f21fb1b-d8db-4c02-b153-22cdb89c3251" (UID: "8f21fb1b-d8db-4c02-b153-22cdb89c3251"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:35.197424 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.197385    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f21fb1b-d8db-4c02-b153-22cdb89c3251-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:35:35.851931 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.851900    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"
Jul 16 22:35:35.852136 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.851906    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r" event={"ID":"8f21fb1b-d8db-4c02-b153-22cdb89c3251","Type":"ContainerDied","Data":"58eb8ec74fa369282f9e2c6cec6337263e36ecf010c03aa67a3672a3fe6f059e"}
Jul 16 22:35:35.852136 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.852029    2555 scope.go:117] "RemoveContainer" containerID="d0dbe5207d7b6ddb6dff3bb3d257c1f7d51f8fd7afaeb8ff5a5b55d58a96f5e5"
Jul 16 22:35:35.875147 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.875121    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"]
Jul 16 22:35:35.879157 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:35.879134    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-57t7r"]
Jul 16 22:35:36.494638 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:36.494608    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" path="/var/lib/kubelet/pods/8f21fb1b-d8db-4c02-b153-22cdb89c3251/volumes"
Jul 16 22:35:42.489731 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:42.489692    2555 scope.go:117] "RemoveContainer" containerID="3157e711d07e0168afb5191f9c3df9e67722b91d50434781859db2e9e32cf666"
Jul 16 22:35:42.884880 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:42.884849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" event={"ID":"291b1a8d-cc42-40b5-a82e-ad1161d94de5","Type":"ContainerStarted","Data":"e1091a075c605a412238c7a3d271498619336d2cb812bd1f455e5d75d4962406"}
Jul 16 22:35:42.907862 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:42.907808    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" podStartSLOduration=17.907792963 podStartE2EDuration="17.907792963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:42.90703916 +0000 UTC m=+2898.900203314" watchObservedRunningTime="2026-07-16 22:35:42.907792963 +0000 UTC m=+2898.900957115"
Jul 16 22:35:43.891755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:43.891724    2555 generic.go:356] "Generic (PLEG): container finished" podID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerID="e1091a075c605a412238c7a3d271498619336d2cb812bd1f455e5d75d4962406" exitCode=6
Jul 16 22:35:43.892149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:43.891772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" event={"ID":"291b1a8d-cc42-40b5-a82e-ad1161d94de5","Type":"ContainerDied","Data":"e1091a075c605a412238c7a3d271498619336d2cb812bd1f455e5d75d4962406"}
Jul 16 22:35:43.892149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:43.891827    2555 scope.go:117] "RemoveContainer" containerID="3157e711d07e0168afb5191f9c3df9e67722b91d50434781859db2e9e32cf666"
Jul 16 22:35:43.892149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:43.892103    2555 scope.go:117] "RemoveContainer" containerID="e1091a075c605a412238c7a3d271498619336d2cb812bd1f455e5d75d4962406"
Jul 16 22:35:43.892367 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:43.892343    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-5jctf_odh-ai-gateway-infra(291b1a8d-cc42-40b5-a82e-ad1161d94de5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5"
Jul 16 22:35:43.941279 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:43.941242    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"]
Jul 16 22:35:45.029117 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.029095    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"
Jul 16 22:35:45.189031 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.188930    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b1a8d-cc42-40b5-a82e-ad1161d94de5-maas-api-service-ca\") pod \"291b1a8d-cc42-40b5-a82e-ad1161d94de5\" (UID: \"291b1a8d-cc42-40b5-a82e-ad1161d94de5\") "
Jul 16 22:35:45.189374 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.189345    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291b1a8d-cc42-40b5-a82e-ad1161d94de5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "291b1a8d-cc42-40b5-a82e-ad1161d94de5" (UID: "291b1a8d-cc42-40b5-a82e-ad1161d94de5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:45.290508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.290472    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b1a8d-cc42-40b5-a82e-ad1161d94de5-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:35:45.343734 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.343693    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"]
Jul 16 22:35:45.344052 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344037    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344055    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344067    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:45.344097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344072    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:45.344097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344079    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344084    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344091    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344097 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344097    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344111    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:45.344319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344117    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:45.344319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344166    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344173    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:45.344319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344180    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:45.344319 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.344188    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:45.347202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.347178    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"
Jul 16 22:35:45.362217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.362186    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"]
Jul 16 22:35:45.491824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.491735    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7605ccb9-5558-4bb8-86f9-a837f73ebf6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7\" (UID: \"7605ccb9-5558-4bb8-86f9-a837f73ebf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"
Jul 16 22:35:45.593186 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.593147    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7605ccb9-5558-4bb8-86f9-a837f73ebf6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7\" (UID: \"7605ccb9-5558-4bb8-86f9-a837f73ebf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"
Jul 16 22:35:45.593820 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.593801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7605ccb9-5558-4bb8-86f9-a837f73ebf6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7\" (UID: \"7605ccb9-5558-4bb8-86f9-a837f73ebf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"
Jul 16 22:35:45.657017 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.656979    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"
Jul 16 22:35:45.804807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.804570    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"]
Jul 16 22:35:45.807260 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:35:45.807228    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7605ccb9_5558_4bb8_86f9_a837f73ebf6f.slice/crio-639b2c0901cb16b69c928c0325804b6e09873d4a7bbf6587f0e36beb825facda WatchSource:0}: Error finding container 639b2c0901cb16b69c928c0325804b6e09873d4a7bbf6587f0e36beb825facda: Status 404 returned error can't find the container with id 639b2c0901cb16b69c928c0325804b6e09873d4a7bbf6587f0e36beb825facda
Jul 16 22:35:45.901731 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.901682    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7" event={"ID":"7605ccb9-5558-4bb8-86f9-a837f73ebf6f","Type":"ContainerStarted","Data":"639b2c0901cb16b69c928c0325804b6e09873d4a7bbf6587f0e36beb825facda"}
Jul 16 22:35:45.902972 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.902931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf" event={"ID":"291b1a8d-cc42-40b5-a82e-ad1161d94de5","Type":"ContainerDied","Data":"68922e4d85f5b2ed98c155cfaed869225493e9983c5da51948c16d1830c16fc6"}
Jul 16 22:35:45.902972 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.902967    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"
Jul 16 22:35:45.903177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.903007    2555 scope.go:117] "RemoveContainer" containerID="e1091a075c605a412238c7a3d271498619336d2cb812bd1f455e5d75d4962406"
Jul 16 22:35:45.927216 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.927188    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"]
Jul 16 22:35:45.936869 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:45.936840    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-5jctf"]
Jul 16 22:35:46.490243 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:46.490209    2555 scope.go:117] "RemoveContainer" containerID="e9f55265a4fe062d77337d71d1deaf2b593b47e55739c06ccaba696695d6bf26"
Jul 16 22:35:46.495225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:46.495204    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" path="/var/lib/kubelet/pods/291b1a8d-cc42-40b5-a82e-ad1161d94de5/volumes"
Jul 16 22:35:46.908668 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:46.908634    2555 generic.go:356] "Generic (PLEG): container finished" podID="7605ccb9-5558-4bb8-86f9-a837f73ebf6f" containerID="c3a9fed64c9e210ebeb44fbced89d243bd83ee79841ac8b54457ac0ad2048643" exitCode=0
Jul 16 22:35:46.908861 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:46.908724    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7" event={"ID":"7605ccb9-5558-4bb8-86f9-a837f73ebf6f","Type":"ContainerDied","Data":"c3a9fed64c9e210ebeb44fbced89d243bd83ee79841ac8b54457ac0ad2048643"}
Jul 16 22:35:46.910457 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:46.910426    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" event={"ID":"5c6d0b87-6d79-460e-be02-b41aaba37f09","Type":"ContainerStarted","Data":"377cc5163a5e96faca5251674b71dd34dc7ca8965845cee52ceb3b9c079eee34"}
Jul 16 22:35:47.304673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.304566    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" podStartSLOduration=19.304546523 podStartE2EDuration="19.304546523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:46.980666184 +0000 UTC m=+2902.973830338" watchObservedRunningTime="2026-07-16 22:35:47.304546523 +0000 UTC m=+2903.297710717"
Jul 16 22:35:47.304886 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.304871    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"]
Jul 16 22:35:47.305239 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.305223    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:47.305276 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.305242    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:47.305313 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.305301    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="291b1a8d-cc42-40b5-a82e-ad1161d94de5" containerName="revoke-keys"
Jul 16 22:35:47.305353 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.305319    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f21fb1b-d8db-4c02-b153-22cdb89c3251" containerName="revoke-keys"
Jul 16 22:35:47.308218 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.308202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"
Jul 16 22:35:47.320772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.320746    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"]
Jul 16 22:35:47.409291 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.409251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad639c8-a18a-4eb7-9b9c-7682eef82fa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n\" (UID: \"cad639c8-a18a-4eb7-9b9c-7682eef82fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"
Jul 16 22:35:47.510306 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.510265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad639c8-a18a-4eb7-9b9c-7682eef82fa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n\" (UID: \"cad639c8-a18a-4eb7-9b9c-7682eef82fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"
Jul 16 22:35:47.510906 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.510885    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad639c8-a18a-4eb7-9b9c-7682eef82fa5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n\" (UID: \"cad639c8-a18a-4eb7-9b9c-7682eef82fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"
Jul 16 22:35:47.617482 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.617442    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"
Jul 16 22:35:47.750464 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.750431    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"]
Jul 16 22:35:47.753089 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:35:47.753053    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad639c8_a18a_4eb7_9b9c_7682eef82fa5.slice/crio-a125b528a4b704ec277967ca49238b039f2377554f651aba556296df78d38cbc WatchSource:0}: Error finding container a125b528a4b704ec277967ca49238b039f2377554f651aba556296df78d38cbc: Status 404 returned error can't find the container with id a125b528a4b704ec277967ca49238b039f2377554f651aba556296df78d38cbc
Jul 16 22:35:47.916603 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.916508    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" event={"ID":"cad639c8-a18a-4eb7-9b9c-7682eef82fa5","Type":"ContainerStarted","Data":"a125b528a4b704ec277967ca49238b039f2377554f651aba556296df78d38cbc"}
Jul 16 22:35:47.918422 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.918392    2555 generic.go:356] "Generic (PLEG): container finished" podID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerID="377cc5163a5e96faca5251674b71dd34dc7ca8965845cee52ceb3b9c079eee34" exitCode=6
Jul 16 22:35:47.918567 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.918467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" event={"ID":"5c6d0b87-6d79-460e-be02-b41aaba37f09","Type":"ContainerDied","Data":"377cc5163a5e96faca5251674b71dd34dc7ca8965845cee52ceb3b9c079eee34"}
Jul 16 22:35:47.918567 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.918521    2555 scope.go:117] "RemoveContainer" containerID="e9f55265a4fe062d77337d71d1deaf2b593b47e55739c06ccaba696695d6bf26"
Jul 16 22:35:47.918912 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.918896    2555 scope.go:117] "RemoveContainer" containerID="377cc5163a5e96faca5251674b71dd34dc7ca8965845cee52ceb3b9c079eee34"
Jul 16 22:35:47.919198 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:47.919178    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk_odh-ai-gateway-infra(5c6d0b87-6d79-460e-be02-b41aaba37f09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09"
Jul 16 22:35:47.986714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:47.986677    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"]
Jul 16 22:35:48.045068 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.045043    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"
Jul 16 22:35:48.217059 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.216947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7605ccb9-5558-4bb8-86f9-a837f73ebf6f-maas-api-service-ca\") pod \"7605ccb9-5558-4bb8-86f9-a837f73ebf6f\" (UID: \"7605ccb9-5558-4bb8-86f9-a837f73ebf6f\") "
Jul 16 22:35:48.217303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.217280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7605ccb9-5558-4bb8-86f9-a837f73ebf6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7605ccb9-5558-4bb8-86f9-a837f73ebf6f" (UID: "7605ccb9-5558-4bb8-86f9-a837f73ebf6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:48.317671 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.317627    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7605ccb9-5558-4bb8-86f9-a837f73ebf6f-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:35:48.923648 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.923615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7" event={"ID":"7605ccb9-5558-4bb8-86f9-a837f73ebf6f","Type":"ContainerDied","Data":"639b2c0901cb16b69c928c0325804b6e09873d4a7bbf6587f0e36beb825facda"}
Jul 16 22:35:48.924119 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.923668    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="639b2c0901cb16b69c928c0325804b6e09873d4a7bbf6587f0e36beb825facda"
Jul 16 22:35:48.924119 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.923620    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7"
Jul 16 22:35:48.925235 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.925208    2555 generic.go:356] "Generic (PLEG): container finished" podID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerID="10316f8427fd389b527234800cbe784c4b2be310a4e71a4bd1b7e930b668a602" exitCode=6
Jul 16 22:35:48.925357 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.925293    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" event={"ID":"cad639c8-a18a-4eb7-9b9c-7682eef82fa5","Type":"ContainerDied","Data":"10316f8427fd389b527234800cbe784c4b2be310a4e71a4bd1b7e930b668a602"}
Jul 16 22:35:48.925469 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:48.925444    2555 scope.go:117] "RemoveContainer" containerID="10316f8427fd389b527234800cbe784c4b2be310a4e71a4bd1b7e930b668a602"
Jul 16 22:35:49.059932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.059904    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"
Jul 16 22:35:49.225059 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.224977    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d0b87-6d79-460e-be02-b41aaba37f09-maas-api-service-ca\") pod \"5c6d0b87-6d79-460e-be02-b41aaba37f09\" (UID: \"5c6d0b87-6d79-460e-be02-b41aaba37f09\") "
Jul 16 22:35:49.225348 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.225327    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6d0b87-6d79-460e-be02-b41aaba37f09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c6d0b87-6d79-460e-be02-b41aaba37f09" (UID: "5c6d0b87-6d79-460e-be02-b41aaba37f09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:49.326443 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.326406    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d0b87-6d79-460e-be02-b41aaba37f09-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:35:49.933530 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.933495    2555 generic.go:356] "Generic (PLEG): container finished" podID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerID="5e3ea9f150185a03cb0460bf0c891cc44a370f69e01da304c1065f14e5b1627a" exitCode=6
Jul 16 22:35:49.933998 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.933581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" event={"ID":"cad639c8-a18a-4eb7-9b9c-7682eef82fa5","Type":"ContainerDied","Data":"5e3ea9f150185a03cb0460bf0c891cc44a370f69e01da304c1065f14e5b1627a"}
Jul 16 22:35:49.933998 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.933640    2555 scope.go:117] "RemoveContainer" containerID="10316f8427fd389b527234800cbe784c4b2be310a4e71a4bd1b7e930b668a602"
Jul 16 22:35:49.933998 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.933791    2555 scope.go:117] "RemoveContainer" containerID="5e3ea9f150185a03cb0460bf0c891cc44a370f69e01da304c1065f14e5b1627a"
Jul 16 22:35:49.934457 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:49.934067    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n_odh-ai-gateway-infra(cad639c8-a18a-4eb7-9b9c-7682eef82fa5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5"
Jul 16 22:35:49.934977 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.934888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk" event={"ID":"5c6d0b87-6d79-460e-be02-b41aaba37f09","Type":"ContainerDied","Data":"8721d4281a335125ad459770e1b8104a0e2a69c053889fededc0b72a35189cd7"}
Jul 16 22:35:49.934977 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.934898    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"
Jul 16 22:35:49.944812 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.944795    2555 scope.go:117] "RemoveContainer" containerID="377cc5163a5e96faca5251674b71dd34dc7ca8965845cee52ceb3b9c079eee34"
Jul 16 22:35:49.964902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.964877    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"]
Jul 16 22:35:49.969029 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:49.969006    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2hqfk"]
Jul 16 22:35:50.495374 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:50.495336    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" path="/var/lib/kubelet/pods/5c6d0b87-6d79-460e-be02-b41aaba37f09/volumes"
Jul 16 22:35:50.941759 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:35:50.941728    2555 scope.go:117] "RemoveContainer" containerID="5e3ea9f150185a03cb0460bf0c891cc44a370f69e01da304c1065f14e5b1627a"
Jul 16 22:35:50.942247 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:35:50.941990    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n_odh-ai-gateway-infra(cad639c8-a18a-4eb7-9b9c-7682eef82fa5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5"
Jul 16 22:36:06.160019 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.159982    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"]
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160279    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160289    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160298    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160304    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160312    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7605ccb9-5558-4bb8-86f9-a837f73ebf6f" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160317    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7605ccb9-5558-4bb8-86f9-a837f73ebf6f" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160324    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160329    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160375    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160382    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7605ccb9-5558-4bb8-86f9-a837f73ebf6f" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160389    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.160508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.160395    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6d0b87-6d79-460e-be02-b41aaba37f09" containerName="revoke-keys"
Jul 16 22:36:06.163192 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.163173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"
Jul 16 22:36:06.170426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.169807    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"]
Jul 16 22:36:06.274629 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.274587    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc11e5-c075-4c04-8c61-6ca7a019ca43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864\" (UID: \"a8bc11e5-c075-4c04-8c61-6ca7a019ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"
Jul 16 22:36:06.375578 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.375537    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc11e5-c075-4c04-8c61-6ca7a019ca43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864\" (UID: \"a8bc11e5-c075-4c04-8c61-6ca7a019ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"
Jul 16 22:36:06.376245 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.376221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc11e5-c075-4c04-8c61-6ca7a019ca43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864\" (UID: \"a8bc11e5-c075-4c04-8c61-6ca7a019ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"
Jul 16 22:36:06.474714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.474632    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"
Jul 16 22:36:06.489887 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.489858    2555 scope.go:117] "RemoveContainer" containerID="5e3ea9f150185a03cb0460bf0c891cc44a370f69e01da304c1065f14e5b1627a"
Jul 16 22:36:06.618019 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:36:06.617980    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bc11e5_c075_4c04_8c61_6ca7a019ca43.slice/crio-0baa54f742841f1a1bd8b0d0919dccd80ee45675c7b75f0897394817e1765d20 WatchSource:0}: Error finding container 0baa54f742841f1a1bd8b0d0919dccd80ee45675c7b75f0897394817e1765d20: Status 404 returned error can't find the container with id 0baa54f742841f1a1bd8b0d0919dccd80ee45675c7b75f0897394817e1765d20
Jul 16 22:36:06.618177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:06.618131    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"]
Jul 16 22:36:07.014494 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:07.014397    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" event={"ID":"a8bc11e5-c075-4c04-8c61-6ca7a019ca43","Type":"ContainerStarted","Data":"8f26e12e8093d6be9911c2d6bf843009e8668c0aa1b1806207e40b22cf219a17"}
Jul 16 22:36:07.014494 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:07.014455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" event={"ID":"a8bc11e5-c075-4c04-8c61-6ca7a019ca43","Type":"ContainerStarted","Data":"0baa54f742841f1a1bd8b0d0919dccd80ee45675c7b75f0897394817e1765d20"}
Jul 16 22:36:07.016131 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:07.016108    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" event={"ID":"cad639c8-a18a-4eb7-9b9c-7682eef82fa5","Type":"ContainerStarted","Data":"ac9c0a22b9826b37bb65e6d49f53bf3751136942ec7b5618bc0d921918b05315"}
Jul 16 22:36:07.032678 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:07.032635    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" podStartSLOduration=1.032620392 podStartE2EDuration="1.032620392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:07.031604481 +0000 UTC m=+2923.024768633" watchObservedRunningTime="2026-07-16 22:36:07.032620392 +0000 UTC m=+2923.025784544"
Jul 16 22:36:08.022094 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.022058    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerID="8f26e12e8093d6be9911c2d6bf843009e8668c0aa1b1806207e40b22cf219a17" exitCode=6
Jul 16 22:36:08.022568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.022151    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" event={"ID":"a8bc11e5-c075-4c04-8c61-6ca7a019ca43","Type":"ContainerDied","Data":"8f26e12e8093d6be9911c2d6bf843009e8668c0aa1b1806207e40b22cf219a17"}
Jul 16 22:36:08.022568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.022350    2555 scope.go:117] "RemoveContainer" containerID="8f26e12e8093d6be9911c2d6bf843009e8668c0aa1b1806207e40b22cf219a17"
Jul 16 22:36:08.024133 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.024111    2555 generic.go:356] "Generic (PLEG): container finished" podID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerID="ac9c0a22b9826b37bb65e6d49f53bf3751136942ec7b5618bc0d921918b05315" exitCode=6
Jul 16 22:36:08.024240 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.024169    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" event={"ID":"cad639c8-a18a-4eb7-9b9c-7682eef82fa5","Type":"ContainerDied","Data":"ac9c0a22b9826b37bb65e6d49f53bf3751136942ec7b5618bc0d921918b05315"}
Jul 16 22:36:08.024240 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.024212    2555 scope.go:117] "RemoveContainer" containerID="5e3ea9f150185a03cb0460bf0c891cc44a370f69e01da304c1065f14e5b1627a"
Jul 16 22:36:08.024506 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.024490    2555 scope.go:117] "RemoveContainer" containerID="ac9c0a22b9826b37bb65e6d49f53bf3751136942ec7b5618bc0d921918b05315"
Jul 16 22:36:08.024735 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:36:08.024680    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n_odh-ai-gateway-infra(cad639c8-a18a-4eb7-9b9c-7682eef82fa5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5"
Jul 16 22:36:08.040699 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.040637    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" podStartSLOduration=21.040616363 podStartE2EDuration="21.040616363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:07.054049275 +0000 UTC m=+2923.047213439" watchObservedRunningTime="2026-07-16 22:36:08.040616363 +0000 UTC m=+2924.033780516"
Jul 16 22:36:08.068313 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:08.068281    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"]
Jul 16 22:36:09.031217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.031122    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerID="9c9c180119572322e6f4319501e33a880d785154574d613317815c0b46011c4d" exitCode=6
Jul 16 22:36:09.031217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.031198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" event={"ID":"a8bc11e5-c075-4c04-8c61-6ca7a019ca43","Type":"ContainerDied","Data":"9c9c180119572322e6f4319501e33a880d785154574d613317815c0b46011c4d"}
Jul 16 22:36:09.031728 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.031255    2555 scope.go:117] "RemoveContainer" containerID="8f26e12e8093d6be9911c2d6bf843009e8668c0aa1b1806207e40b22cf219a17"
Jul 16 22:36:09.031728 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.031436    2555 scope.go:117] "RemoveContainer" containerID="9c9c180119572322e6f4319501e33a880d785154574d613317815c0b46011c4d"
Jul 16 22:36:09.031728 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:36:09.031650    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864_odh-ai-gateway-infra(a8bc11e5-c075-4c04-8c61-6ca7a019ca43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43"
Jul 16 22:36:09.162736 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.162709    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"
Jul 16 22:36:09.200177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.200140    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad639c8-a18a-4eb7-9b9c-7682eef82fa5-maas-api-service-ca\") pod \"cad639c8-a18a-4eb7-9b9c-7682eef82fa5\" (UID: \"cad639c8-a18a-4eb7-9b9c-7682eef82fa5\") "
Jul 16 22:36:09.200470 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.200449    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad639c8-a18a-4eb7-9b9c-7682eef82fa5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cad639c8-a18a-4eb7-9b9c-7682eef82fa5" (UID: "cad639c8-a18a-4eb7-9b9c-7682eef82fa5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:09.301157 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:09.301064    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad639c8-a18a-4eb7-9b9c-7682eef82fa5-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:36:10.040230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:10.040195    2555 scope.go:117] "RemoveContainer" containerID="9c9c180119572322e6f4319501e33a880d785154574d613317815c0b46011c4d"
Jul 16 22:36:10.040670 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:36:10.040417    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864_odh-ai-gateway-infra(a8bc11e5-c075-4c04-8c61-6ca7a019ca43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43"
Jul 16 22:36:10.041442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:10.041419    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n" event={"ID":"cad639c8-a18a-4eb7-9b9c-7682eef82fa5","Type":"ContainerDied","Data":"a125b528a4b704ec277967ca49238b039f2377554f651aba556296df78d38cbc"}
Jul 16 22:36:10.041583 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:10.041427    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"
Jul 16 22:36:10.041583 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:10.041473    2555 scope.go:117] "RemoveContainer" containerID="ac9c0a22b9826b37bb65e6d49f53bf3751136942ec7b5618bc0d921918b05315"
Jul 16 22:36:10.072236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:10.072210    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"]
Jul 16 22:36:10.076147 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:10.076122    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-6bk5n"]
Jul 16 22:36:10.494768 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:10.494738    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" path="/var/lib/kubelet/pods/cad639c8-a18a-4eb7-9b9c-7682eef82fa5/volumes"
Jul 16 22:36:11.462256 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.462214    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"]
Jul 16 22:36:11.462755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.462589    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:11.462755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.462608    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:11.462755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.462619    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:11.462755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.462626    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:11.462755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.462680    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:11.462755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.462694    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:11.466904 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.466884    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"
Jul 16 22:36:11.475562 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.475527    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"]
Jul 16 22:36:11.517318 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.517283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7\" (UID: \"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"
Jul 16 22:36:11.618788 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.618745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7\" (UID: \"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"
Jul 16 22:36:11.619397 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.619377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7\" (UID: \"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"
Jul 16 22:36:11.779935 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.779842    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"
Jul 16 22:36:11.909830 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:11.909803    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"]
Jul 16 22:36:11.912708 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:36:11.912680    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afa8c6d_ad4b_4405_b4ce_9773ad1e12ed.slice/crio-03c7ebbf0d67fbab5c28c250cad8f8578682211c91f164901b9f2338d3a031bf WatchSource:0}: Error finding container 03c7ebbf0d67fbab5c28c250cad8f8578682211c91f164901b9f2338d3a031bf: Status 404 returned error can't find the container with id 03c7ebbf0d67fbab5c28c250cad8f8578682211c91f164901b9f2338d3a031bf
Jul 16 22:36:12.053256 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:12.053175    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7" event={"ID":"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed","Type":"ContainerStarted","Data":"03c7ebbf0d67fbab5c28c250cad8f8578682211c91f164901b9f2338d3a031bf"}
Jul 16 22:36:13.059349 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:13.059262    2555 generic.go:356] "Generic (PLEG): container finished" podID="8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed" containerID="e25b35417e07607e92349bfa2cc4eee906566fef6926c0e54384b01f63287514" exitCode=0
Jul 16 22:36:13.059705 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:13.059357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7" event={"ID":"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed","Type":"ContainerDied","Data":"e25b35417e07607e92349bfa2cc4eee906566fef6926c0e54384b01f63287514"}
Jul 16 22:36:14.192848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:14.192822    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"
Jul 16 22:36:14.240227 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:14.240186    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed-maas-api-service-ca\") pod \"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed\" (UID: \"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed\") "
Jul 16 22:36:14.240499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:14.240478    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed" (UID: "8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:14.341683 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:14.341651    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:36:15.069521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:15.069494    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7"
Jul 16 22:36:15.069521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:15.069519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7" event={"ID":"8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed","Type":"ContainerDied","Data":"03c7ebbf0d67fbab5c28c250cad8f8578682211c91f164901b9f2338d3a031bf"}
Jul 16 22:36:15.069764 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:15.069561    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c7ebbf0d67fbab5c28c250cad8f8578682211c91f164901b9f2338d3a031bf"
Jul 16 22:36:23.489924 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:23.489888    2555 scope.go:117] "RemoveContainer" containerID="9c9c180119572322e6f4319501e33a880d785154574d613317815c0b46011c4d"
Jul 16 22:36:24.110624 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:24.110590    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" event={"ID":"a8bc11e5-c075-4c04-8c61-6ca7a019ca43","Type":"ContainerStarted","Data":"a2cba40fbbd57f093c49070fcec4bda5f077ee5d7753decc909a8c8850723794"}
Jul 16 22:36:25.116315 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:25.116283    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerID="a2cba40fbbd57f093c49070fcec4bda5f077ee5d7753decc909a8c8850723794" exitCode=6
Jul 16 22:36:25.116755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:25.116339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" event={"ID":"a8bc11e5-c075-4c04-8c61-6ca7a019ca43","Type":"ContainerDied","Data":"a2cba40fbbd57f093c49070fcec4bda5f077ee5d7753decc909a8c8850723794"}
Jul 16 22:36:25.116755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:25.116384    2555 scope.go:117] "RemoveContainer" containerID="9c9c180119572322e6f4319501e33a880d785154574d613317815c0b46011c4d"
Jul 16 22:36:25.116755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:25.116668    2555 scope.go:117] "RemoveContainer" containerID="a2cba40fbbd57f093c49070fcec4bda5f077ee5d7753decc909a8c8850723794"
Jul 16 22:36:25.116916 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:36:25.116870    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864_odh-ai-gateway-infra(a8bc11e5-c075-4c04-8c61-6ca7a019ca43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43"
Jul 16 22:36:25.144670 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:25.144629    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"]
Jul 16 22:36:26.251442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:26.251416    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"
Jul 16 22:36:26.358796 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:26.358753    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc11e5-c075-4c04-8c61-6ca7a019ca43-maas-api-service-ca\") pod \"a8bc11e5-c075-4c04-8c61-6ca7a019ca43\" (UID: \"a8bc11e5-c075-4c04-8c61-6ca7a019ca43\") "
Jul 16 22:36:26.359158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:26.359132    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bc11e5-c075-4c04-8c61-6ca7a019ca43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8bc11e5-c075-4c04-8c61-6ca7a019ca43" (UID: "a8bc11e5-c075-4c04-8c61-6ca7a019ca43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:26.460382 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:26.460299    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc11e5-c075-4c04-8c61-6ca7a019ca43-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:36:27.126704 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:27.126672    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"
Jul 16 22:36:27.126893 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:27.126674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864" event={"ID":"a8bc11e5-c075-4c04-8c61-6ca7a019ca43","Type":"ContainerDied","Data":"0baa54f742841f1a1bd8b0d0919dccd80ee45675c7b75f0897394817e1765d20"}
Jul 16 22:36:27.126893 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:27.126804    2555 scope.go:117] "RemoveContainer" containerID="a2cba40fbbd57f093c49070fcec4bda5f077ee5d7753decc909a8c8850723794"
Jul 16 22:36:27.144684 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:27.144659    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"]
Jul 16 22:36:27.152605 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:27.152577    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-r9864"]
Jul 16 22:36:28.494333 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:28.494302    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" path="/var/lib/kubelet/pods/a8bc11e5-c075-4c04-8c61-6ca7a019ca43/volumes"
Jul 16 22:36:47.385104 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385029    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"]
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385339    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385350    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385365    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385371    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385379    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385384    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385397    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385403    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385450    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad639c8-a18a-4eb7-9b9c-7682eef82fa5" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385459    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385464    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385471    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed" containerName="revoke-keys"
Jul 16 22:36:47.385557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.385477    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:36:47.388449 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.388434    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"
Jul 16 22:36:47.398282 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.398263    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:36:47.399913 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.399894    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:36:47.414352 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.414323    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"]
Jul 16 22:36:47.437875 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.437839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d930d4-955c-47ce-a308-2fcd7582f21c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m\" (UID: \"37d930d4-955c-47ce-a308-2fcd7582f21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"
Jul 16 22:36:47.538987 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.538925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d930d4-955c-47ce-a308-2fcd7582f21c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m\" (UID: \"37d930d4-955c-47ce-a308-2fcd7582f21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"
Jul 16 22:36:47.539559 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.539543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d930d4-955c-47ce-a308-2fcd7582f21c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m\" (UID: \"37d930d4-955c-47ce-a308-2fcd7582f21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"
Jul 16 22:36:47.698102 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.698015    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"
Jul 16 22:36:47.839217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:47.839109    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"]
Jul 16 22:36:47.841338 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:36:47.841305    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d930d4_955c_47ce_a308_2fcd7582f21c.slice/crio-e30681097fe46aa35a2dae7bd708656e28cadfb9bc6f256cc8f2ca6c7ad8c682 WatchSource:0}: Error finding container e30681097fe46aa35a2dae7bd708656e28cadfb9bc6f256cc8f2ca6c7ad8c682: Status 404 returned error can't find the container with id e30681097fe46aa35a2dae7bd708656e28cadfb9bc6f256cc8f2ca6c7ad8c682
Jul 16 22:36:48.220929 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:48.220882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" event={"ID":"37d930d4-955c-47ce-a308-2fcd7582f21c","Type":"ContainerStarted","Data":"d45c281e92629ef0c914d175980b355ff1f99a075f3e8ec57e77d08c149b52b1"}
Jul 16 22:36:48.221141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:48.220948    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" event={"ID":"37d930d4-955c-47ce-a308-2fcd7582f21c","Type":"ContainerStarted","Data":"e30681097fe46aa35a2dae7bd708656e28cadfb9bc6f256cc8f2ca6c7ad8c682"}
Jul 16 22:36:49.225903 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:49.225866    2555 generic.go:356] "Generic (PLEG): container finished" podID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerID="d45c281e92629ef0c914d175980b355ff1f99a075f3e8ec57e77d08c149b52b1" exitCode=6
Jul 16 22:36:49.226322 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:49.225978    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" event={"ID":"37d930d4-955c-47ce-a308-2fcd7582f21c","Type":"ContainerDied","Data":"d45c281e92629ef0c914d175980b355ff1f99a075f3e8ec57e77d08c149b52b1"}
Jul 16 22:36:49.226322 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:49.226148    2555 scope.go:117] "RemoveContainer" containerID="d45c281e92629ef0c914d175980b355ff1f99a075f3e8ec57e77d08c149b52b1"
Jul 16 22:36:50.232834 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:50.232802    2555 generic.go:356] "Generic (PLEG): container finished" podID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerID="bde2b12f4836aecebf8573e697af0b96ec890b2512dcd593deb596c53e34a2a1" exitCode=6
Jul 16 22:36:50.233310 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:50.232887    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" event={"ID":"37d930d4-955c-47ce-a308-2fcd7582f21c","Type":"ContainerDied","Data":"bde2b12f4836aecebf8573e697af0b96ec890b2512dcd593deb596c53e34a2a1"}
Jul 16 22:36:50.233310 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:50.232949    2555 scope.go:117] "RemoveContainer" containerID="d45c281e92629ef0c914d175980b355ff1f99a075f3e8ec57e77d08c149b52b1"
Jul 16 22:36:50.233310 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:50.233121    2555 scope.go:117] "RemoveContainer" containerID="bde2b12f4836aecebf8573e697af0b96ec890b2512dcd593deb596c53e34a2a1"
Jul 16 22:36:50.233465 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:36:50.233359    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m_odh-ai-gateway-infra(37d930d4-955c-47ce-a308-2fcd7582f21c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c"
Jul 16 22:36:51.238463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:36:51.238429    2555 scope.go:117] "RemoveContainer" containerID="bde2b12f4836aecebf8573e697af0b96ec890b2512dcd593deb596c53e34a2a1"
Jul 16 22:36:51.238844 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:36:51.238669    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m_odh-ai-gateway-infra(37d930d4-955c-47ce-a308-2fcd7582f21c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c"
Jul 16 22:37:02.052044 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.052012    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"]
Jul 16 22:37:02.052448 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.052357    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:37:02.052448 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.052374    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bc11e5-c075-4c04-8c61-6ca7a019ca43" containerName="revoke-keys"
Jul 16 22:37:02.056468 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.056448    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"
Jul 16 22:37:02.062834 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.062808    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"]
Jul 16 22:37:02.159013 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.158973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72801518-24a8-4126-8b4d-6665ae3dc9c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5\" (UID: \"72801518-24a8-4126-8b4d-6665ae3dc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"
Jul 16 22:37:02.259427 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.259391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72801518-24a8-4126-8b4d-6665ae3dc9c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5\" (UID: \"72801518-24a8-4126-8b4d-6665ae3dc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"
Jul 16 22:37:02.260036 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.260015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72801518-24a8-4126-8b4d-6665ae3dc9c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5\" (UID: \"72801518-24a8-4126-8b4d-6665ae3dc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"
Jul 16 22:37:02.367775 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.367742    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"
Jul 16 22:37:02.505798 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:02.505775    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"]
Jul 16 22:37:02.507998 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:37:02.507971    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72801518_24a8_4126_8b4d_6665ae3dc9c8.slice/crio-ca0130c353b2a952dcc68793f427c8556cbffdcd3f094af790630b2535b708b2 WatchSource:0}: Error finding container ca0130c353b2a952dcc68793f427c8556cbffdcd3f094af790630b2535b708b2: Status 404 returned error can't find the container with id ca0130c353b2a952dcc68793f427c8556cbffdcd3f094af790630b2535b708b2
Jul 16 22:37:03.293286 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:03.293253    2555 generic.go:356] "Generic (PLEG): container finished" podID="72801518-24a8-4126-8b4d-6665ae3dc9c8" containerID="6bfa15d262a432cfb0fba8786e5a060778336b1ffbd4a39f14ac679ac5c79c30" exitCode=0
Jul 16 22:37:03.293653 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:03.293331    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5" event={"ID":"72801518-24a8-4126-8b4d-6665ae3dc9c8","Type":"ContainerDied","Data":"6bfa15d262a432cfb0fba8786e5a060778336b1ffbd4a39f14ac679ac5c79c30"}
Jul 16 22:37:03.293653 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:03.293385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5" event={"ID":"72801518-24a8-4126-8b4d-6665ae3dc9c8","Type":"ContainerStarted","Data":"ca0130c353b2a952dcc68793f427c8556cbffdcd3f094af790630b2535b708b2"}
Jul 16 22:37:03.490181 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:03.490108    2555 scope.go:117] "RemoveContainer" containerID="bde2b12f4836aecebf8573e697af0b96ec890b2512dcd593deb596c53e34a2a1"
Jul 16 22:37:04.299893 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.299859    2555 generic.go:356] "Generic (PLEG): container finished" podID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerID="cc6f0de74d84de26b3b0dc5a280f96976f1a7047b8a23a00c93e4fcb01db3f44" exitCode=6
Jul 16 22:37:04.300371 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.299936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" event={"ID":"37d930d4-955c-47ce-a308-2fcd7582f21c","Type":"ContainerDied","Data":"cc6f0de74d84de26b3b0dc5a280f96976f1a7047b8a23a00c93e4fcb01db3f44"}
Jul 16 22:37:04.300371 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.300010    2555 scope.go:117] "RemoveContainer" containerID="bde2b12f4836aecebf8573e697af0b96ec890b2512dcd593deb596c53e34a2a1"
Jul 16 22:37:04.300371 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.300353    2555 scope.go:117] "RemoveContainer" containerID="cc6f0de74d84de26b3b0dc5a280f96976f1a7047b8a23a00c93e4fcb01db3f44"
Jul 16 22:37:04.300782 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:04.300597    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m_odh-ai-gateway-infra(37d930d4-955c-47ce-a308-2fcd7582f21c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c"
Jul 16 22:37:04.431407 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.431379    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"
Jul 16 22:37:04.476294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.476261    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72801518-24a8-4126-8b4d-6665ae3dc9c8-maas-api-service-ca\") pod \"72801518-24a8-4126-8b4d-6665ae3dc9c8\" (UID: \"72801518-24a8-4126-8b4d-6665ae3dc9c8\") "
Jul 16 22:37:04.476643 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.476619    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72801518-24a8-4126-8b4d-6665ae3dc9c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72801518-24a8-4126-8b4d-6665ae3dc9c8" (UID: "72801518-24a8-4126-8b4d-6665ae3dc9c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:04.576840 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:04.576803    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72801518-24a8-4126-8b4d-6665ae3dc9c8-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:37:05.306807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.306761    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5" event={"ID":"72801518-24a8-4126-8b4d-6665ae3dc9c8","Type":"ContainerDied","Data":"ca0130c353b2a952dcc68793f427c8556cbffdcd3f094af790630b2535b708b2"}
Jul 16 22:37:05.306807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.306785    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5"
Jul 16 22:37:05.307355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.306842    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0130c353b2a952dcc68793f427c8556cbffdcd3f094af790630b2535b708b2"
Jul 16 22:37:05.330888 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.330858    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"]
Jul 16 22:37:05.459690 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.459667    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"
Jul 16 22:37:05.486168 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.486139    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d930d4-955c-47ce-a308-2fcd7582f21c-maas-api-service-ca\") pod \"37d930d4-955c-47ce-a308-2fcd7582f21c\" (UID: \"37d930d4-955c-47ce-a308-2fcd7582f21c\") "
Jul 16 22:37:05.486496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.486474    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d930d4-955c-47ce-a308-2fcd7582f21c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37d930d4-955c-47ce-a308-2fcd7582f21c" (UID: "37d930d4-955c-47ce-a308-2fcd7582f21c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:05.586707 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:05.586679    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d930d4-955c-47ce-a308-2fcd7582f21c-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:37:06.249367 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249326    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"]
Jul 16 22:37:06.249654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249640    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:06.249699 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249658    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:06.249699 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249686    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:06.249699 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249692    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:06.249824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249701    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72801518-24a8-4126-8b4d-6665ae3dc9c8" containerName="revoke-keys"
Jul 16 22:37:06.249824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249715    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="72801518-24a8-4126-8b4d-6665ae3dc9c8" containerName="revoke-keys"
Jul 16 22:37:06.249824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249775    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="72801518-24a8-4126-8b4d-6665ae3dc9c8" containerName="revoke-keys"
Jul 16 22:37:06.249824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249785    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:06.249824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249791    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:06.249824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.249800    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:06.252671 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.252656    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"
Jul 16 22:37:06.266988 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.266943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"]
Jul 16 22:37:06.292277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.292244    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a185504-eeeb-44ed-a635-793db56aae60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs\" (UID: \"4a185504-eeeb-44ed-a635-793db56aae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"
Jul 16 22:37:06.311935 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.311904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m" event={"ID":"37d930d4-955c-47ce-a308-2fcd7582f21c","Type":"ContainerDied","Data":"e30681097fe46aa35a2dae7bd708656e28cadfb9bc6f256cc8f2ca6c7ad8c682"}
Jul 16 22:37:06.312394 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.311942    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"
Jul 16 22:37:06.312394 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.311986    2555 scope.go:117] "RemoveContainer" containerID="cc6f0de74d84de26b3b0dc5a280f96976f1a7047b8a23a00c93e4fcb01db3f44"
Jul 16 22:37:06.335095 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.335069    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"]
Jul 16 22:37:06.344768 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.344738    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-5gn4m"]
Jul 16 22:37:06.394552 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.394519    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a185504-eeeb-44ed-a635-793db56aae60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs\" (UID: \"4a185504-eeeb-44ed-a635-793db56aae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"
Jul 16 22:37:06.395162 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.395144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a185504-eeeb-44ed-a635-793db56aae60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs\" (UID: \"4a185504-eeeb-44ed-a635-793db56aae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"
Jul 16 22:37:06.494930 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.494896    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" path="/var/lib/kubelet/pods/37d930d4-955c-47ce-a308-2fcd7582f21c/volumes"
Jul 16 22:37:06.561932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.561845    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"
Jul 16 22:37:06.689089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:06.689007    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"]
Jul 16 22:37:06.691057 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:37:06.691031    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a185504_eeeb_44ed_a635_793db56aae60.slice/crio-23f8a9d9f483dba119fe0ae89cf88875b7f4ff7b2202f823da5cede2737835fc WatchSource:0}: Error finding container 23f8a9d9f483dba119fe0ae89cf88875b7f4ff7b2202f823da5cede2737835fc: Status 404 returned error can't find the container with id 23f8a9d9f483dba119fe0ae89cf88875b7f4ff7b2202f823da5cede2737835fc
Jul 16 22:37:07.318845 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:07.318810    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a185504-eeeb-44ed-a635-793db56aae60" containerID="444bf19c094175c0c7746edd8a9f312e5546bd320205005dab9af90cf4744df9" exitCode=6
Jul 16 22:37:07.319270 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:07.318860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" event={"ID":"4a185504-eeeb-44ed-a635-793db56aae60","Type":"ContainerDied","Data":"444bf19c094175c0c7746edd8a9f312e5546bd320205005dab9af90cf4744df9"}
Jul 16 22:37:07.319270 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:07.318922    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" event={"ID":"4a185504-eeeb-44ed-a635-793db56aae60","Type":"ContainerStarted","Data":"23f8a9d9f483dba119fe0ae89cf88875b7f4ff7b2202f823da5cede2737835fc"}
Jul 16 22:37:07.319270 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:07.319112    2555 scope.go:117] "RemoveContainer" containerID="444bf19c094175c0c7746edd8a9f312e5546bd320205005dab9af90cf4744df9"
Jul 16 22:37:08.325146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:08.325053    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a185504-eeeb-44ed-a635-793db56aae60" containerID="15e7d84e1e01a92492c0f90299f4848f7cc82b34d6af10db4ef4e72e73f9c75d" exitCode=6
Jul 16 22:37:08.325539 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:08.325143    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" event={"ID":"4a185504-eeeb-44ed-a635-793db56aae60","Type":"ContainerDied","Data":"15e7d84e1e01a92492c0f90299f4848f7cc82b34d6af10db4ef4e72e73f9c75d"}
Jul 16 22:37:08.325539 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:08.325200    2555 scope.go:117] "RemoveContainer" containerID="444bf19c094175c0c7746edd8a9f312e5546bd320205005dab9af90cf4744df9"
Jul 16 22:37:08.325539 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:08.325357    2555 scope.go:117] "RemoveContainer" containerID="15e7d84e1e01a92492c0f90299f4848f7cc82b34d6af10db4ef4e72e73f9c75d"
Jul 16 22:37:08.325684 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:08.325582    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs_odh-ai-gateway-infra(4a185504-eeeb-44ed-a635-793db56aae60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" podUID="4a185504-eeeb-44ed-a635-793db56aae60"
Jul 16 22:37:09.331444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:09.331410    2555 scope.go:117] "RemoveContainer" containerID="15e7d84e1e01a92492c0f90299f4848f7cc82b34d6af10db4ef4e72e73f9c75d"
Jul 16 22:37:09.331853 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:09.331608    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs_odh-ai-gateway-infra(4a185504-eeeb-44ed-a635-793db56aae60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" podUID="4a185504-eeeb-44ed-a635-793db56aae60"
Jul 16 22:37:15.655772 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.655736    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"]
Jul 16 22:37:15.656176 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.656095    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:15.656176 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.656108    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d930d4-955c-47ce-a308-2fcd7582f21c" containerName="revoke-keys"
Jul 16 22:37:15.660338 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.660320    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"
Jul 16 22:37:15.667135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.667110    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"]
Jul 16 22:37:15.774735 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.774696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec36f1-0d06-4395-949c-79ac1215313d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-9b264\" (UID: \"b4ec36f1-0d06-4395-949c-79ac1215313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"
Jul 16 22:37:15.875342 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.875306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec36f1-0d06-4395-949c-79ac1215313d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-9b264\" (UID: \"b4ec36f1-0d06-4395-949c-79ac1215313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"
Jul 16 22:37:15.875893 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.875876    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec36f1-0d06-4395-949c-79ac1215313d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-9b264\" (UID: \"b4ec36f1-0d06-4395-949c-79ac1215313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"
Jul 16 22:37:15.972503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:15.972416    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"
Jul 16 22:37:16.309254 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:16.309229    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"]
Jul 16 22:37:16.311479 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:37:16.311455    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ec36f1_0d06_4395_949c_79ac1215313d.slice/crio-676d13db7c416a3b1712ff9f2885b34846c578bb0ef3100783b5e2ac57641011 WatchSource:0}: Error finding container 676d13db7c416a3b1712ff9f2885b34846c578bb0ef3100783b5e2ac57641011: Status 404 returned error can't find the container with id 676d13db7c416a3b1712ff9f2885b34846c578bb0ef3100783b5e2ac57641011
Jul 16 22:37:16.364646 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:16.364615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" event={"ID":"b4ec36f1-0d06-4395-949c-79ac1215313d","Type":"ContainerStarted","Data":"676d13db7c416a3b1712ff9f2885b34846c578bb0ef3100783b5e2ac57641011"}
Jul 16 22:37:17.370374 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:17.370332    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerID="d957375068b4c17b44a25a7012e44680ac5a0de25dfadf22528cb0346c83f79e" exitCode=6
Jul 16 22:37:17.370374 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:17.370377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" event={"ID":"b4ec36f1-0d06-4395-949c-79ac1215313d","Type":"ContainerDied","Data":"d957375068b4c17b44a25a7012e44680ac5a0de25dfadf22528cb0346c83f79e"}
Jul 16 22:37:17.370850 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:17.370606    2555 scope.go:117] "RemoveContainer" containerID="d957375068b4c17b44a25a7012e44680ac5a0de25dfadf22528cb0346c83f79e"
Jul 16 22:37:18.376928 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:18.376894    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerID="dad08c49230a1bf9ebed1b8900e4d1d82c576c27293277c9110dae3a11b4d10d" exitCode=6
Jul 16 22:37:18.377546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:18.376989    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" event={"ID":"b4ec36f1-0d06-4395-949c-79ac1215313d","Type":"ContainerDied","Data":"dad08c49230a1bf9ebed1b8900e4d1d82c576c27293277c9110dae3a11b4d10d"}
Jul 16 22:37:18.377546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:18.377048    2555 scope.go:117] "RemoveContainer" containerID="d957375068b4c17b44a25a7012e44680ac5a0de25dfadf22528cb0346c83f79e"
Jul 16 22:37:18.377546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:18.377290    2555 scope.go:117] "RemoveContainer" containerID="dad08c49230a1bf9ebed1b8900e4d1d82c576c27293277c9110dae3a11b4d10d"
Jul 16 22:37:18.377546 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:18.377521    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-9b264_odh-ai-gateway-infra(b4ec36f1-0d06-4395-949c-79ac1215313d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d"
Jul 16 22:37:19.382887 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:19.382859    2555 scope.go:117] "RemoveContainer" containerID="dad08c49230a1bf9ebed1b8900e4d1d82c576c27293277c9110dae3a11b4d10d"
Jul 16 22:37:19.383295 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:19.383070    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-9b264_odh-ai-gateway-infra(b4ec36f1-0d06-4395-949c-79ac1215313d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d"
Jul 16 22:37:24.492547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:24.492509    2555 scope.go:117] "RemoveContainer" containerID="15e7d84e1e01a92492c0f90299f4848f7cc82b34d6af10db4ef4e72e73f9c75d"
Jul 16 22:37:25.412572 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:25.412539    2555 generic.go:356] "Generic (PLEG): container finished" podID="4a185504-eeeb-44ed-a635-793db56aae60" containerID="1f15d162fabe65bc136fd9ae53dc31206d238c8b7210188a3683f777453582e8" exitCode=6
Jul 16 22:37:25.412752 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:25.412608    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" event={"ID":"4a185504-eeeb-44ed-a635-793db56aae60","Type":"ContainerDied","Data":"1f15d162fabe65bc136fd9ae53dc31206d238c8b7210188a3683f777453582e8"}
Jul 16 22:37:25.412752 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:25.412669    2555 scope.go:117] "RemoveContainer" containerID="15e7d84e1e01a92492c0f90299f4848f7cc82b34d6af10db4ef4e72e73f9c75d"
Jul 16 22:37:25.413013 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:25.412992    2555 scope.go:117] "RemoveContainer" containerID="1f15d162fabe65bc136fd9ae53dc31206d238c8b7210188a3683f777453582e8"
Jul 16 22:37:25.413245 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:25.413226    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs_odh-ai-gateway-infra(4a185504-eeeb-44ed-a635-793db56aae60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" podUID="4a185504-eeeb-44ed-a635-793db56aae60"
Jul 16 22:37:26.438574 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:26.438537    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"]
Jul 16 22:37:26.576414 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:26.576388    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"
Jul 16 22:37:26.669930 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:26.669889    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a185504-eeeb-44ed-a635-793db56aae60-maas-api-service-ca\") pod \"4a185504-eeeb-44ed-a635-793db56aae60\" (UID: \"4a185504-eeeb-44ed-a635-793db56aae60\") "
Jul 16 22:37:26.670272 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:26.670248    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a185504-eeeb-44ed-a635-793db56aae60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a185504-eeeb-44ed-a635-793db56aae60" (UID: "4a185504-eeeb-44ed-a635-793db56aae60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:26.771040 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:26.770920    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a185504-eeeb-44ed-a635-793db56aae60-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:37:27.424149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:27.424115    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"
Jul 16 22:37:27.424149 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:27.424137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs" event={"ID":"4a185504-eeeb-44ed-a635-793db56aae60","Type":"ContainerDied","Data":"23f8a9d9f483dba119fe0ae89cf88875b7f4ff7b2202f823da5cede2737835fc"}
Jul 16 22:37:27.424392 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:27.424192    2555 scope.go:117] "RemoveContainer" containerID="1f15d162fabe65bc136fd9ae53dc31206d238c8b7210188a3683f777453582e8"
Jul 16 22:37:27.446852 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:27.446827    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"]
Jul 16 22:37:27.452785 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:27.452760    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-zc6hs"]
Jul 16 22:37:28.203185 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.203153    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"]
Jul 16 22:37:28.203458 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.203446    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:28.203511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.203460    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:28.203511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.203473    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:28.203511 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.203478    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:28.203613 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.203534    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:28.203613 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.203546    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:28.207741 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.207725    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"
Jul 16 22:37:28.216003 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.215979    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"]
Jul 16 22:37:28.283212 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.283175    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2302143-f76f-4432-9cad-2aa49fd25f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s\" (UID: \"f2302143-f76f-4432-9cad-2aa49fd25f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"
Jul 16 22:37:28.384478 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.384442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2302143-f76f-4432-9cad-2aa49fd25f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s\" (UID: \"f2302143-f76f-4432-9cad-2aa49fd25f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"
Jul 16 22:37:28.385167 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.385141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2302143-f76f-4432-9cad-2aa49fd25f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s\" (UID: \"f2302143-f76f-4432-9cad-2aa49fd25f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"
Jul 16 22:37:28.495217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.495118    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a185504-eeeb-44ed-a635-793db56aae60" path="/var/lib/kubelet/pods/4a185504-eeeb-44ed-a635-793db56aae60/volumes"
Jul 16 22:37:28.518975 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.518914    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"
Jul 16 22:37:28.862378 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:28.862349    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"]
Jul 16 22:37:28.864251 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:37:28.864229    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2302143_f76f_4432_9cad_2aa49fd25f68.slice/crio-9ded7d9732849e74e8ff71d44a73f3ee4e440ae396819a9ffacefe287315c408 WatchSource:0}: Error finding container 9ded7d9732849e74e8ff71d44a73f3ee4e440ae396819a9ffacefe287315c408: Status 404 returned error can't find the container with id 9ded7d9732849e74e8ff71d44a73f3ee4e440ae396819a9ffacefe287315c408
Jul 16 22:37:29.435036 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:29.435003    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s" event={"ID":"f2302143-f76f-4432-9cad-2aa49fd25f68","Type":"ContainerStarted","Data":"6ca2844efff4db6d4035fc55191a94d0b2b5ffeffbbde5f928384371c964bc4a"}
Jul 16 22:37:29.435036 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:29.435052    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s" event={"ID":"f2302143-f76f-4432-9cad-2aa49fd25f68","Type":"ContainerStarted","Data":"9ded7d9732849e74e8ff71d44a73f3ee4e440ae396819a9ffacefe287315c408"}
Jul 16 22:37:29.463123 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:29.463069    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s" podStartSLOduration=1.463053704 podStartE2EDuration="1.463053704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:29.46258076 +0000 UTC m=+3005.455744912" watchObservedRunningTime="2026-07-16 22:37:29.463053704 +0000 UTC m=+3005.456217856"
Jul 16 22:37:30.441509 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:30.441470    2555 generic.go:356] "Generic (PLEG): container finished" podID="f2302143-f76f-4432-9cad-2aa49fd25f68" containerID="6ca2844efff4db6d4035fc55191a94d0b2b5ffeffbbde5f928384371c964bc4a" exitCode=0
Jul 16 22:37:30.441923 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:30.441536    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s" event={"ID":"f2302143-f76f-4432-9cad-2aa49fd25f68","Type":"ContainerDied","Data":"6ca2844efff4db6d4035fc55191a94d0b2b5ffeffbbde5f928384371c964bc4a"}
Jul 16 22:37:31.490188 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:31.490158    2555 scope.go:117] "RemoveContainer" containerID="dad08c49230a1bf9ebed1b8900e4d1d82c576c27293277c9110dae3a11b4d10d"
Jul 16 22:37:31.578912 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:31.578886    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"
Jul 16 22:37:31.713246 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:31.713148    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2302143-f76f-4432-9cad-2aa49fd25f68-maas-api-service-ca\") pod \"f2302143-f76f-4432-9cad-2aa49fd25f68\" (UID: \"f2302143-f76f-4432-9cad-2aa49fd25f68\") "
Jul 16 22:37:31.713596 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:31.713566    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2302143-f76f-4432-9cad-2aa49fd25f68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2302143-f76f-4432-9cad-2aa49fd25f68" (UID: "f2302143-f76f-4432-9cad-2aa49fd25f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:31.813975 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:31.813921    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2302143-f76f-4432-9cad-2aa49fd25f68-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:37:32.452013 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:32.451980    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s"
Jul 16 22:37:32.452223 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:32.452011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s" event={"ID":"f2302143-f76f-4432-9cad-2aa49fd25f68","Type":"ContainerDied","Data":"9ded7d9732849e74e8ff71d44a73f3ee4e440ae396819a9ffacefe287315c408"}
Jul 16 22:37:32.452223 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:32.452089    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ded7d9732849e74e8ff71d44a73f3ee4e440ae396819a9ffacefe287315c408"
Jul 16 22:37:32.453839 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:32.453807    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerID="b821580db5fbc53b742577af7e64d21ac8160bbb7c0a6476e77033e1ece36297" exitCode=6
Jul 16 22:37:32.453951 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:32.453877    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" event={"ID":"b4ec36f1-0d06-4395-949c-79ac1215313d","Type":"ContainerDied","Data":"b821580db5fbc53b742577af7e64d21ac8160bbb7c0a6476e77033e1ece36297"}
Jul 16 22:37:32.453951 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:32.453926    2555 scope.go:117] "RemoveContainer" containerID="dad08c49230a1bf9ebed1b8900e4d1d82c576c27293277c9110dae3a11b4d10d"
Jul 16 22:37:32.454357 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:32.454340    2555 scope.go:117] "RemoveContainer" containerID="b821580db5fbc53b742577af7e64d21ac8160bbb7c0a6476e77033e1ece36297"
Jul 16 22:37:32.454591 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:32.454568    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-9b264_odh-ai-gateway-infra(b4ec36f1-0d06-4395-949c-79ac1215313d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d"
Jul 16 22:37:33.484433 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:33.484399    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"]
Jul 16 22:37:33.609355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:33.609332    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"
Jul 16 22:37:33.731283 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:33.731249    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec36f1-0d06-4395-949c-79ac1215313d-maas-api-service-ca\") pod \"b4ec36f1-0d06-4395-949c-79ac1215313d\" (UID: \"b4ec36f1-0d06-4395-949c-79ac1215313d\") "
Jul 16 22:37:33.731625 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:33.731601    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ec36f1-0d06-4395-949c-79ac1215313d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4ec36f1-0d06-4395-949c-79ac1215313d" (UID: "b4ec36f1-0d06-4395-949c-79ac1215313d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:33.832132 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:33.832093    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec36f1-0d06-4395-949c-79ac1215313d-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:37:34.466039 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:34.466005    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"
Jul 16 22:37:34.466324 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:34.466014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264" event={"ID":"b4ec36f1-0d06-4395-949c-79ac1215313d","Type":"ContainerDied","Data":"676d13db7c416a3b1712ff9f2885b34846c578bb0ef3100783b5e2ac57641011"}
Jul 16 22:37:34.466324 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:34.466138    2555 scope.go:117] "RemoveContainer" containerID="b821580db5fbc53b742577af7e64d21ac8160bbb7c0a6476e77033e1ece36297"
Jul 16 22:37:34.488020 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:34.487994    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"]
Jul 16 22:37:34.496331 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:34.496309    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-9b264"]
Jul 16 22:37:36.494591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:36.494557    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" path="/var/lib/kubelet/pods/b4ec36f1-0d06-4395-949c-79ac1215313d/volumes"
Jul 16 22:37:37.409429 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409393    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"]
Jul 16 22:37:37.409832 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409817    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:37:37.409907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409835    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:37:37.409907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409861    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2302143-f76f-4432-9cad-2aa49fd25f68" containerName="revoke-keys"
Jul 16 22:37:37.409907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409870    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2302143-f76f-4432-9cad-2aa49fd25f68" containerName="revoke-keys"
Jul 16 22:37:37.409907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409883    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:37:37.409907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409892    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:37:37.409907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409902    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:37.409907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409910    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:37.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.409994    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:37:37.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.410005    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:37:37.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.410014    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2302143-f76f-4432-9cad-2aa49fd25f68" containerName="revoke-keys"
Jul 16 22:37:37.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.410023    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a185504-eeeb-44ed-a635-793db56aae60" containerName="revoke-keys"
Jul 16 22:37:37.410202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.410032    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:37:37.414655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.414638    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"
Jul 16 22:37:37.417468 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.417442    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:37:37.417586 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.417442    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:37:37.421111 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.421071    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"]
Jul 16 22:37:37.565170 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.565128    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c88ada-bb52-4d92-8a6f-f1745d3a50c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66\" (UID: \"32c88ada-bb52-4d92-8a6f-f1745d3a50c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"
Jul 16 22:37:37.666079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.665982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c88ada-bb52-4d92-8a6f-f1745d3a50c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66\" (UID: \"32c88ada-bb52-4d92-8a6f-f1745d3a50c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"
Jul 16 22:37:37.666594 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.666573    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c88ada-bb52-4d92-8a6f-f1745d3a50c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66\" (UID: \"32c88ada-bb52-4d92-8a6f-f1745d3a50c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"
Jul 16 22:37:37.727079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.727045    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"
Jul 16 22:37:37.855323 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:37.855299    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"]
Jul 16 22:37:37.857587 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:37:37.857562    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c88ada_bb52_4d92_8a6f_f1745d3a50c4.slice/crio-2c73d00e73069e60b18ee7ad0fe4ddcd01efac0028df41f5721c2fe434fe7694 WatchSource:0}: Error finding container 2c73d00e73069e60b18ee7ad0fe4ddcd01efac0028df41f5721c2fe434fe7694: Status 404 returned error can't find the container with id 2c73d00e73069e60b18ee7ad0fe4ddcd01efac0028df41f5721c2fe434fe7694
Jul 16 22:37:38.486158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:38.486123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" event={"ID":"32c88ada-bb52-4d92-8a6f-f1745d3a50c4","Type":"ContainerStarted","Data":"12a1139ef620fecb1ebaa7b9e71b398ee73cb31402b0a5d88b795c96c7a6f400"}
Jul 16 22:37:38.486307 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:38.486173    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" event={"ID":"32c88ada-bb52-4d92-8a6f-f1745d3a50c4","Type":"ContainerStarted","Data":"2c73d00e73069e60b18ee7ad0fe4ddcd01efac0028df41f5721c2fe434fe7694"}
Jul 16 22:37:38.505006 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:38.504941    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" podStartSLOduration=1.5049271659999999 podStartE2EDuration="1.504927166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:38.502555284 +0000 UTC m=+3014.495719436" watchObservedRunningTime="2026-07-16 22:37:38.504927166 +0000 UTC m=+3014.498091318"
Jul 16 22:37:39.492111 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:39.492078    2555 generic.go:356] "Generic (PLEG): container finished" podID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerID="12a1139ef620fecb1ebaa7b9e71b398ee73cb31402b0a5d88b795c96c7a6f400" exitCode=6
Jul 16 22:37:39.492560 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:39.492146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" event={"ID":"32c88ada-bb52-4d92-8a6f-f1745d3a50c4","Type":"ContainerDied","Data":"12a1139ef620fecb1ebaa7b9e71b398ee73cb31402b0a5d88b795c96c7a6f400"}
Jul 16 22:37:39.492560 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:39.492307    2555 scope.go:117] "RemoveContainer" containerID="12a1139ef620fecb1ebaa7b9e71b398ee73cb31402b0a5d88b795c96c7a6f400"
Jul 16 22:37:40.498171 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:40.498084    2555 generic.go:356] "Generic (PLEG): container finished" podID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerID="76c21edc4e6239c00f5d4db2e100994cd7eeaece0f654328586466f5c9e7e5f5" exitCode=6
Jul 16 22:37:40.498551 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:40.498162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" event={"ID":"32c88ada-bb52-4d92-8a6f-f1745d3a50c4","Type":"ContainerDied","Data":"76c21edc4e6239c00f5d4db2e100994cd7eeaece0f654328586466f5c9e7e5f5"}
Jul 16 22:37:40.498551 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:40.498220    2555 scope.go:117] "RemoveContainer" containerID="12a1139ef620fecb1ebaa7b9e71b398ee73cb31402b0a5d88b795c96c7a6f400"
Jul 16 22:37:40.498551 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:40.498454    2555 scope.go:117] "RemoveContainer" containerID="76c21edc4e6239c00f5d4db2e100994cd7eeaece0f654328586466f5c9e7e5f5"
Jul 16 22:37:40.498687 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:40.498667    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66_odh-ai-gateway-infra(32c88ada-bb52-4d92-8a6f-f1745d3a50c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4"
Jul 16 22:37:41.503972 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:41.503926    2555 scope.go:117] "RemoveContainer" containerID="76c21edc4e6239c00f5d4db2e100994cd7eeaece0f654328586466f5c9e7e5f5"
Jul 16 22:37:41.504390 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:41.504140    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66_odh-ai-gateway-infra(32c88ada-bb52-4d92-8a6f-f1745d3a50c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4"
Jul 16 22:37:53.489863 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:53.489830    2555 scope.go:117] "RemoveContainer" containerID="76c21edc4e6239c00f5d4db2e100994cd7eeaece0f654328586466f5c9e7e5f5"
Jul 16 22:37:54.562372 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:54.562338    2555 generic.go:356] "Generic (PLEG): container finished" podID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerID="e66c5d858852eeefdd516b6d5c2a5ce15fd6c15dd6477e1ee974787e47e4bf07" exitCode=6
Jul 16 22:37:54.562804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:54.562413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" event={"ID":"32c88ada-bb52-4d92-8a6f-f1745d3a50c4","Type":"ContainerDied","Data":"e66c5d858852eeefdd516b6d5c2a5ce15fd6c15dd6477e1ee974787e47e4bf07"}
Jul 16 22:37:54.562804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:54.562482    2555 scope.go:117] "RemoveContainer" containerID="76c21edc4e6239c00f5d4db2e100994cd7eeaece0f654328586466f5c9e7e5f5"
Jul 16 22:37:54.562939 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:54.562822    2555 scope.go:117] "RemoveContainer" containerID="e66c5d858852eeefdd516b6d5c2a5ce15fd6c15dd6477e1ee974787e47e4bf07"
Jul 16 22:37:54.563039 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:37:54.563020    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66_odh-ai-gateway-infra(32c88ada-bb52-4d92-8a6f-f1745d3a50c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4"
Jul 16 22:37:55.612330 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:55.612296    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"]
Jul 16 22:37:55.746895 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:55.746872    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"
Jul 16 22:37:55.822194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:55.822157    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c88ada-bb52-4d92-8a6f-f1745d3a50c4-maas-api-service-ca\") pod \"32c88ada-bb52-4d92-8a6f-f1745d3a50c4\" (UID: \"32c88ada-bb52-4d92-8a6f-f1745d3a50c4\") "
Jul 16 22:37:55.822503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:55.822477    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c88ada-bb52-4d92-8a6f-f1745d3a50c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32c88ada-bb52-4d92-8a6f-f1745d3a50c4" (UID: "32c88ada-bb52-4d92-8a6f-f1745d3a50c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:55.922991 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:55.922893    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c88ada-bb52-4d92-8a6f-f1745d3a50c4-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:37:56.573769 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:56.573740    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"
Jul 16 22:37:56.573984 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:56.573773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66" event={"ID":"32c88ada-bb52-4d92-8a6f-f1745d3a50c4","Type":"ContainerDied","Data":"2c73d00e73069e60b18ee7ad0fe4ddcd01efac0028df41f5721c2fe434fe7694"}
Jul 16 22:37:56.573984 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:56.573835    2555 scope.go:117] "RemoveContainer" containerID="e66c5d858852eeefdd516b6d5c2a5ce15fd6c15dd6477e1ee974787e47e4bf07"
Jul 16 22:37:56.600505 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:56.600482    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"]
Jul 16 22:37:56.612932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:56.612909    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-wjc66"]
Jul 16 22:37:58.494384 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:37:58.494347    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" path="/var/lib/kubelet/pods/32c88ada-bb52-4d92-8a6f-f1745d3a50c4/volumes"
Jul 16 22:38:05.709088 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709055    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"]
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709350    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709360    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709375    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709381    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ec36f1-0d06-4395-949c-79ac1215313d" containerName="revoke-keys"
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709387    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709393    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709445    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:05.709555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.709454    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:05.712165 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.712150    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"
Jul 16 22:38:05.716534 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.716511    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:38:05.716653 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.716511    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:38:05.720717 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.720691    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"]
Jul 16 22:38:05.806553 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.806508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60974fa-3cda-4074-b52a-20792c0d863b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs\" (UID: \"b60974fa-3cda-4074-b52a-20792c0d863b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"
Jul 16 22:38:05.908025 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.907987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60974fa-3cda-4074-b52a-20792c0d863b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs\" (UID: \"b60974fa-3cda-4074-b52a-20792c0d863b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"
Jul 16 22:38:05.908614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:05.908594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60974fa-3cda-4074-b52a-20792c0d863b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs\" (UID: \"b60974fa-3cda-4074-b52a-20792c0d863b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"
Jul 16 22:38:06.023098 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.023004    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"
Jul 16 22:38:06.170745 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.170722    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"]
Jul 16 22:38:06.173553 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:38:06.173525    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60974fa_3cda_4074_b52a_20792c0d863b.slice/crio-22aeb6b6ad9b5951491969aa8890e18f6fb58db6257fd30b10aaf433c3e197ef WatchSource:0}: Error finding container 22aeb6b6ad9b5951491969aa8890e18f6fb58db6257fd30b10aaf433c3e197ef: Status 404 returned error can't find the container with id 22aeb6b6ad9b5951491969aa8890e18f6fb58db6257fd30b10aaf433c3e197ef
Jul 16 22:38:06.363890 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.363865    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"]
Jul 16 22:38:06.364205 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.364193    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:06.364247 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.364207    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:06.364286 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.364253    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c88ada-bb52-4d92-8a6f-f1745d3a50c4" containerName="revoke-keys"
Jul 16 22:38:06.367128 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.367112    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"
Jul 16 22:38:06.386197 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.386171    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"]
Jul 16 22:38:06.412475 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.412442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9aef1-9339-4dd5-85f4-05d306e1c802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5\" (UID: \"a1a9aef1-9339-4dd5-85f4-05d306e1c802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"
Jul 16 22:38:06.513412 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.513382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9aef1-9339-4dd5-85f4-05d306e1c802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5\" (UID: \"a1a9aef1-9339-4dd5-85f4-05d306e1c802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"
Jul 16 22:38:06.514041 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.514023    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9aef1-9339-4dd5-85f4-05d306e1c802-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5\" (UID: \"a1a9aef1-9339-4dd5-85f4-05d306e1c802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"
Jul 16 22:38:06.619543 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.619448    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" event={"ID":"b60974fa-3cda-4074-b52a-20792c0d863b","Type":"ContainerStarted","Data":"d29d0287b1d99aaed5733e9626570e32edda9ee9049d59cec0e4a60f10679a18"}
Jul 16 22:38:06.619543 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.619497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" event={"ID":"b60974fa-3cda-4074-b52a-20792c0d863b","Type":"ContainerStarted","Data":"22aeb6b6ad9b5951491969aa8890e18f6fb58db6257fd30b10aaf433c3e197ef"}
Jul 16 22:38:06.643616 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.643558    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" podStartSLOduration=1.6435436110000001 podStartE2EDuration="1.643543611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:06.642796114 +0000 UTC m=+3042.635960267" watchObservedRunningTime="2026-07-16 22:38:06.643543611 +0000 UTC m=+3042.636707762"
Jul 16 22:38:06.676839 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:06.676803    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"
Jul 16 22:38:07.021084 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:07.020905    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"]
Jul 16 22:38:07.024327 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:38:07.024298    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a9aef1_9339_4dd5_85f4_05d306e1c802.slice/crio-bdb5be2b3b2d9ef728871cdaa91fffddc1897d228ce4c43279887013c36c98b5 WatchSource:0}: Error finding container bdb5be2b3b2d9ef728871cdaa91fffddc1897d228ce4c43279887013c36c98b5: Status 404 returned error can't find the container with id bdb5be2b3b2d9ef728871cdaa91fffddc1897d228ce4c43279887013c36c98b5
Jul 16 22:38:07.625658 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:07.625620    2555 generic.go:356] "Generic (PLEG): container finished" podID="b60974fa-3cda-4074-b52a-20792c0d863b" containerID="d29d0287b1d99aaed5733e9626570e32edda9ee9049d59cec0e4a60f10679a18" exitCode=6
Jul 16 22:38:07.625873 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:07.625714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" event={"ID":"b60974fa-3cda-4074-b52a-20792c0d863b","Type":"ContainerDied","Data":"d29d0287b1d99aaed5733e9626570e32edda9ee9049d59cec0e4a60f10679a18"}
Jul 16 22:38:07.625941 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:07.625876    2555 scope.go:117] "RemoveContainer" containerID="d29d0287b1d99aaed5733e9626570e32edda9ee9049d59cec0e4a60f10679a18"
Jul 16 22:38:07.627274 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:07.627254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" event={"ID":"a1a9aef1-9339-4dd5-85f4-05d306e1c802","Type":"ContainerStarted","Data":"f1de15970f8a2c6f331529a86f4cb1650674abe04fbd983d1213f6c78d6cdabb"}
Jul 16 22:38:07.627377 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:07.627289    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" event={"ID":"a1a9aef1-9339-4dd5-85f4-05d306e1c802","Type":"ContainerStarted","Data":"bdb5be2b3b2d9ef728871cdaa91fffddc1897d228ce4c43279887013c36c98b5"}
Jul 16 22:38:07.721636 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:07.721581    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" podStartSLOduration=1.721564078 podStartE2EDuration="1.721564078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:07.720215788 +0000 UTC m=+3043.713379941" watchObservedRunningTime="2026-07-16 22:38:07.721564078 +0000 UTC m=+3043.714728248"
Jul 16 22:38:08.634430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.634397    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerID="f1de15970f8a2c6f331529a86f4cb1650674abe04fbd983d1213f6c78d6cdabb" exitCode=6
Jul 16 22:38:08.634869 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.634494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" event={"ID":"a1a9aef1-9339-4dd5-85f4-05d306e1c802","Type":"ContainerDied","Data":"f1de15970f8a2c6f331529a86f4cb1650674abe04fbd983d1213f6c78d6cdabb"}
Jul 16 22:38:08.634869 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.634646    2555 scope.go:117] "RemoveContainer" containerID="f1de15970f8a2c6f331529a86f4cb1650674abe04fbd983d1213f6c78d6cdabb"
Jul 16 22:38:08.636448 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.636424    2555 generic.go:356] "Generic (PLEG): container finished" podID="b60974fa-3cda-4074-b52a-20792c0d863b" containerID="d5e5343c994ae896cbbb171c74e1b31cd8ed69a8caefb81dbb1360490f3fff9d" exitCode=6
Jul 16 22:38:08.636556 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.636502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" event={"ID":"b60974fa-3cda-4074-b52a-20792c0d863b","Type":"ContainerDied","Data":"d5e5343c994ae896cbbb171c74e1b31cd8ed69a8caefb81dbb1360490f3fff9d"}
Jul 16 22:38:08.636608 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.636553    2555 scope.go:117] "RemoveContainer" containerID="d29d0287b1d99aaed5733e9626570e32edda9ee9049d59cec0e4a60f10679a18"
Jul 16 22:38:08.636739 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.636718    2555 scope.go:117] "RemoveContainer" containerID="d5e5343c994ae896cbbb171c74e1b31cd8ed69a8caefb81dbb1360490f3fff9d"
Jul 16 22:38:08.637031 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:08.636902    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs_odh-ai-gateway-infra(b60974fa-3cda-4074-b52a-20792c0d863b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" podUID="b60974fa-3cda-4074-b52a-20792c0d863b"
Jul 16 22:38:08.746398 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.746362    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"]
Jul 16 22:38:08.750894 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.750871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"
Jul 16 22:38:08.767176 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.767153    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"]
Jul 16 22:38:08.833433 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.833399    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a9611d-c416-4e9f-bfa9-411b17d1ba25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq\" (UID: \"24a9611d-c416-4e9f-bfa9-411b17d1ba25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"
Jul 16 22:38:08.934489 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.934406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a9611d-c416-4e9f-bfa9-411b17d1ba25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq\" (UID: \"24a9611d-c416-4e9f-bfa9-411b17d1ba25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"
Jul 16 22:38:08.935071 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:08.935053    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a9611d-c416-4e9f-bfa9-411b17d1ba25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq\" (UID: \"24a9611d-c416-4e9f-bfa9-411b17d1ba25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"
Jul 16 22:38:09.061725 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.061689    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"
Jul 16 22:38:09.195343 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.195320    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"]
Jul 16 22:38:09.197637 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:38:09.197599    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a9611d_c416_4e9f_bfa9_411b17d1ba25.slice/crio-c4bf9c650932458bfb2e67f2b32d26a4df190f418c46922f3db133d5d886e57c WatchSource:0}: Error finding container c4bf9c650932458bfb2e67f2b32d26a4df190f418c46922f3db133d5d886e57c: Status 404 returned error can't find the container with id c4bf9c650932458bfb2e67f2b32d26a4df190f418c46922f3db133d5d886e57c
Jul 16 22:38:09.644174 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.644131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" event={"ID":"24a9611d-c416-4e9f-bfa9-411b17d1ba25","Type":"ContainerStarted","Data":"ea6025647f62f6b9aae97ea282c681c12ac75c022279fa76e4770110b9a805f1"}
Jul 16 22:38:09.644650 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.644196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" event={"ID":"24a9611d-c416-4e9f-bfa9-411b17d1ba25","Type":"ContainerStarted","Data":"c4bf9c650932458bfb2e67f2b32d26a4df190f418c46922f3db133d5d886e57c"}
Jul 16 22:38:09.645942 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.645920    2555 scope.go:117] "RemoveContainer" containerID="d5e5343c994ae896cbbb171c74e1b31cd8ed69a8caefb81dbb1360490f3fff9d"
Jul 16 22:38:09.646141 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:09.646117    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs_odh-ai-gateway-infra(b60974fa-3cda-4074-b52a-20792c0d863b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" podUID="b60974fa-3cda-4074-b52a-20792c0d863b"
Jul 16 22:38:09.647620 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.647592    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerID="9ec52938a6f6484b9129b112b4d8c327dcba2737e58a6b254e190cdf9cdcee2e" exitCode=6
Jul 16 22:38:09.647767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.647642    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" event={"ID":"a1a9aef1-9339-4dd5-85f4-05d306e1c802","Type":"ContainerDied","Data":"9ec52938a6f6484b9129b112b4d8c327dcba2737e58a6b254e190cdf9cdcee2e"}
Jul 16 22:38:09.647767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.647686    2555 scope.go:117] "RemoveContainer" containerID="f1de15970f8a2c6f331529a86f4cb1650674abe04fbd983d1213f6c78d6cdabb"
Jul 16 22:38:09.647889 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.647855    2555 scope.go:117] "RemoveContainer" containerID="9ec52938a6f6484b9129b112b4d8c327dcba2737e58a6b254e190cdf9cdcee2e"
Jul 16 22:38:09.648086 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:09.648062    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5_odh-ai-gateway-infra(a1a9aef1-9339-4dd5-85f4-05d306e1c802)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802"
Jul 16 22:38:09.660533 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:09.660494    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" podStartSLOduration=1.660479514 podStartE2EDuration="1.660479514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:09.659602916 +0000 UTC m=+3045.652767068" watchObservedRunningTime="2026-07-16 22:38:09.660479514 +0000 UTC m=+3045.653643660"
Jul 16 22:38:10.660194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:10.660165    2555 scope.go:117] "RemoveContainer" containerID="9ec52938a6f6484b9129b112b4d8c327dcba2737e58a6b254e190cdf9cdcee2e"
Jul 16 22:38:10.660632 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:10.660353    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5_odh-ai-gateway-infra(a1a9aef1-9339-4dd5-85f4-05d306e1c802)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802"
Jul 16 22:38:10.661364 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:10.661336    2555 generic.go:356] "Generic (PLEG): container finished" podID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerID="ea6025647f62f6b9aae97ea282c681c12ac75c022279fa76e4770110b9a805f1" exitCode=6
Jul 16 22:38:10.661465 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:10.661378    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" event={"ID":"24a9611d-c416-4e9f-bfa9-411b17d1ba25","Type":"ContainerDied","Data":"ea6025647f62f6b9aae97ea282c681c12ac75c022279fa76e4770110b9a805f1"}
Jul 16 22:38:10.661570 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:10.661559    2555 scope.go:117] "RemoveContainer" containerID="ea6025647f62f6b9aae97ea282c681c12ac75c022279fa76e4770110b9a805f1"
Jul 16 22:38:11.669419 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:11.669363    2555 generic.go:356] "Generic (PLEG): container finished" podID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerID="874e11bb2145f34771d477d0a3938d658514b6322d2e349ce37b118cfb7ed245" exitCode=6
Jul 16 22:38:11.669908 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:11.669528    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" event={"ID":"24a9611d-c416-4e9f-bfa9-411b17d1ba25","Type":"ContainerDied","Data":"874e11bb2145f34771d477d0a3938d658514b6322d2e349ce37b118cfb7ed245"}
Jul 16 22:38:11.669908 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:11.669671    2555 scope.go:117] "RemoveContainer" containerID="ea6025647f62f6b9aae97ea282c681c12ac75c022279fa76e4770110b9a805f1"
Jul 16 22:38:11.670312 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:11.670289    2555 scope.go:117] "RemoveContainer" containerID="874e11bb2145f34771d477d0a3938d658514b6322d2e349ce37b118cfb7ed245"
Jul 16 22:38:11.670673 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:11.670645    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq_odh-ai-gateway-infra(24a9611d-c416-4e9f-bfa9-411b17d1ba25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25"
Jul 16 22:38:12.675682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:12.675653    2555 scope.go:117] "RemoveContainer" containerID="874e11bb2145f34771d477d0a3938d658514b6322d2e349ce37b118cfb7ed245"
Jul 16 22:38:12.676114 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:12.675830    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq_odh-ai-gateway-infra(24a9611d-c416-4e9f-bfa9-411b17d1ba25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25"
Jul 16 22:38:22.490460 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:22.490370    2555 scope.go:117] "RemoveContainer" containerID="d5e5343c994ae896cbbb171c74e1b31cd8ed69a8caefb81dbb1360490f3fff9d"
Jul 16 22:38:22.726496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:22.726466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" event={"ID":"b60974fa-3cda-4074-b52a-20792c0d863b","Type":"ContainerStarted","Data":"096618b0fc96457b33a22ab1c86e2e45c95e1b6b740219425ea721715633ed6e"}
Jul 16 22:38:23.733079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:23.733045    2555 generic.go:356] "Generic (PLEG): container finished" podID="b60974fa-3cda-4074-b52a-20792c0d863b" containerID="096618b0fc96457b33a22ab1c86e2e45c95e1b6b740219425ea721715633ed6e" exitCode=6
Jul 16 22:38:23.733494 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:23.733115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" event={"ID":"b60974fa-3cda-4074-b52a-20792c0d863b","Type":"ContainerDied","Data":"096618b0fc96457b33a22ab1c86e2e45c95e1b6b740219425ea721715633ed6e"}
Jul 16 22:38:23.733494 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:23.733180    2555 scope.go:117] "RemoveContainer" containerID="d5e5343c994ae896cbbb171c74e1b31cd8ed69a8caefb81dbb1360490f3fff9d"
Jul 16 22:38:23.733494 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:23.733489    2555 scope.go:117] "RemoveContainer" containerID="096618b0fc96457b33a22ab1c86e2e45c95e1b6b740219425ea721715633ed6e"
Jul 16 22:38:23.733703 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:23.733677    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs_odh-ai-gateway-infra(b60974fa-3cda-4074-b52a-20792c0d863b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" podUID="b60974fa-3cda-4074-b52a-20792c0d863b"
Jul 16 22:38:23.759862 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:23.759826    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"]
Jul 16 22:38:24.492382 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:24.492356    2555 scope.go:117] "RemoveContainer" containerID="9ec52938a6f6484b9129b112b4d8c327dcba2737e58a6b254e190cdf9cdcee2e"
Jul 16 22:38:24.875396 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:24.875376    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"
Jul 16 22:38:24.978804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:24.978762    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60974fa-3cda-4074-b52a-20792c0d863b-maas-api-service-ca\") pod \"b60974fa-3cda-4074-b52a-20792c0d863b\" (UID: \"b60974fa-3cda-4074-b52a-20792c0d863b\") "
Jul 16 22:38:24.979115 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:24.979090    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60974fa-3cda-4074-b52a-20792c0d863b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b60974fa-3cda-4074-b52a-20792c0d863b" (UID: "b60974fa-3cda-4074-b52a-20792c0d863b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:25.079854 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.079822    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60974fa-3cda-4074-b52a-20792c0d863b-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:38:25.746568 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.746531    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerID="623094ba0496fa837141d167b2823844376d71c545fda89ddce4aad24f7a80e5" exitCode=6
Jul 16 22:38:25.746763 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.746599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" event={"ID":"a1a9aef1-9339-4dd5-85f4-05d306e1c802","Type":"ContainerDied","Data":"623094ba0496fa837141d167b2823844376d71c545fda89ddce4aad24f7a80e5"}
Jul 16 22:38:25.746763 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.746657    2555 scope.go:117] "RemoveContainer" containerID="9ec52938a6f6484b9129b112b4d8c327dcba2737e58a6b254e190cdf9cdcee2e"
Jul 16 22:38:25.746971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.746938    2555 scope.go:117] "RemoveContainer" containerID="623094ba0496fa837141d167b2823844376d71c545fda89ddce4aad24f7a80e5"
Jul 16 22:38:25.747213 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:25.747167    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5_odh-ai-gateway-infra(a1a9aef1-9339-4dd5-85f4-05d306e1c802)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802"
Jul 16 22:38:25.747991 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.747971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs" event={"ID":"b60974fa-3cda-4074-b52a-20792c0d863b","Type":"ContainerDied","Data":"22aeb6b6ad9b5951491969aa8890e18f6fb58db6257fd30b10aaf433c3e197ef"}
Jul 16 22:38:25.748064 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.748010    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"
Jul 16 22:38:25.757638 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.757624    2555 scope.go:117] "RemoveContainer" containerID="096618b0fc96457b33a22ab1c86e2e45c95e1b6b740219425ea721715633ed6e"
Jul 16 22:38:25.777145 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.777125    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"]
Jul 16 22:38:25.781333 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:25.781313    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gsjjs"]
Jul 16 22:38:26.490296 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:26.490270    2555 scope.go:117] "RemoveContainer" containerID="874e11bb2145f34771d477d0a3938d658514b6322d2e349ce37b118cfb7ed245"
Jul 16 22:38:26.495559 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:26.495529    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" path="/var/lib/kubelet/pods/b60974fa-3cda-4074-b52a-20792c0d863b/volumes"
Jul 16 22:38:26.753581 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:26.753551    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" event={"ID":"24a9611d-c416-4e9f-bfa9-411b17d1ba25","Type":"ContainerStarted","Data":"0351e5a95577c12a4cfbe19124e0229c4c6197c4e6d5845fdc2a9f0da0efdbb7"}
Jul 16 22:38:26.771607 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:26.771581    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"]
Jul 16 22:38:26.900027 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:26.900001    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"
Jul 16 22:38:26.994929 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:26.994847    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9aef1-9339-4dd5-85f4-05d306e1c802-maas-api-service-ca\") pod \"a1a9aef1-9339-4dd5-85f4-05d306e1c802\" (UID: \"a1a9aef1-9339-4dd5-85f4-05d306e1c802\") "
Jul 16 22:38:26.995210 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:26.995187    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a9aef1-9339-4dd5-85f4-05d306e1c802-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1a9aef1-9339-4dd5-85f4-05d306e1c802" (UID: "a1a9aef1-9339-4dd5-85f4-05d306e1c802"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:27.096286 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.096252    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9aef1-9339-4dd5-85f4-05d306e1c802-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:38:27.473784 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.473749    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"]
Jul 16 22:38:27.474074 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474061    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474076    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474086    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474092    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474099    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474104    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474111    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474117    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474135    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:38:27.474141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474141    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:38:27.474418 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474183    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:38:27.474418 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474190    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:38:27.474418 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474198    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474418 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474203    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.474418 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.474210    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" containerName="revoke-keys"
Jul 16 22:38:27.477173 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.477154    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"
Jul 16 22:38:27.487564 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.487535    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"]
Jul 16 22:38:27.599642 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.599607    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a553e9b-dc46-42fb-9f92-f57d0cfbff89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97\" (UID: \"5a553e9b-dc46-42fb-9f92-f57d0cfbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"
Jul 16 22:38:27.700326 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.700286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a553e9b-dc46-42fb-9f92-f57d0cfbff89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97\" (UID: \"5a553e9b-dc46-42fb-9f92-f57d0cfbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"
Jul 16 22:38:27.700923 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.700902    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a553e9b-dc46-42fb-9f92-f57d0cfbff89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97\" (UID: \"5a553e9b-dc46-42fb-9f92-f57d0cfbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"
Jul 16 22:38:27.761809 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.761721    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"
Jul 16 22:38:27.761809 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.761756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5" event={"ID":"a1a9aef1-9339-4dd5-85f4-05d306e1c802","Type":"ContainerDied","Data":"bdb5be2b3b2d9ef728871cdaa91fffddc1897d228ce4c43279887013c36c98b5"}
Jul 16 22:38:27.762084 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.761826    2555 scope.go:117] "RemoveContainer" containerID="623094ba0496fa837141d167b2823844376d71c545fda89ddce4aad24f7a80e5"
Jul 16 22:38:27.763867 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.763835    2555 generic.go:356] "Generic (PLEG): container finished" podID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerID="0351e5a95577c12a4cfbe19124e0229c4c6197c4e6d5845fdc2a9f0da0efdbb7" exitCode=6
Jul 16 22:38:27.764040 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.763867    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" event={"ID":"24a9611d-c416-4e9f-bfa9-411b17d1ba25","Type":"ContainerDied","Data":"0351e5a95577c12a4cfbe19124e0229c4c6197c4e6d5845fdc2a9f0da0efdbb7"}
Jul 16 22:38:27.764246 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.764228    2555 scope.go:117] "RemoveContainer" containerID="0351e5a95577c12a4cfbe19124e0229c4c6197c4e6d5845fdc2a9f0da0efdbb7"
Jul 16 22:38:27.764496 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:27.764474    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq_odh-ai-gateway-infra(24a9611d-c416-4e9f-bfa9-411b17d1ba25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25"
Jul 16 22:38:27.770885 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.770868    2555 scope.go:117] "RemoveContainer" containerID="874e11bb2145f34771d477d0a3938d658514b6322d2e349ce37b118cfb7ed245"
Jul 16 22:38:27.788080 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.788058    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"
Jul 16 22:38:27.795628 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.795599    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"]
Jul 16 22:38:27.801426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.801403    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"]
Jul 16 22:38:27.809880 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.809855    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-h6kg5"]
Jul 16 22:38:27.937722 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:27.937700    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"]
Jul 16 22:38:27.939030 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:38:27.939006    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a553e9b_dc46_42fb_9f92_f57d0cfbff89.slice/crio-74b3725a1feb6857f1f08454a0289b20d14a00d54d8671d84c8fddb637ceeec7 WatchSource:0}: Error finding container 74b3725a1feb6857f1f08454a0289b20d14a00d54d8671d84c8fddb637ceeec7: Status 404 returned error can't find the container with id 74b3725a1feb6857f1f08454a0289b20d14a00d54d8671d84c8fddb637ceeec7
Jul 16 22:38:28.495207 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:28.495130    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a9aef1-9339-4dd5-85f4-05d306e1c802" path="/var/lib/kubelet/pods/a1a9aef1-9339-4dd5-85f4-05d306e1c802/volumes"
Jul 16 22:38:28.771883 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:28.771788    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerID="23e02d8e9f123cc5b13b3f1fdea3be82467747b18d55841ca5ee9a8d393aec5b" exitCode=6
Jul 16 22:38:28.772316 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:28.771876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" event={"ID":"5a553e9b-dc46-42fb-9f92-f57d0cfbff89","Type":"ContainerDied","Data":"23e02d8e9f123cc5b13b3f1fdea3be82467747b18d55841ca5ee9a8d393aec5b"}
Jul 16 22:38:28.772316 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:28.771934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" event={"ID":"5a553e9b-dc46-42fb-9f92-f57d0cfbff89","Type":"ContainerStarted","Data":"74b3725a1feb6857f1f08454a0289b20d14a00d54d8671d84c8fddb637ceeec7"}
Jul 16 22:38:28.772316 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:28.772092    2555 scope.go:117] "RemoveContainer" containerID="23e02d8e9f123cc5b13b3f1fdea3be82467747b18d55841ca5ee9a8d393aec5b"
Jul 16 22:38:28.901106 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:28.901081    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"
Jul 16 22:38:29.011348 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.011317    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a9611d-c416-4e9f-bfa9-411b17d1ba25-maas-api-service-ca\") pod \"24a9611d-c416-4e9f-bfa9-411b17d1ba25\" (UID: \"24a9611d-c416-4e9f-bfa9-411b17d1ba25\") "
Jul 16 22:38:29.011713 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.011689    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a9611d-c416-4e9f-bfa9-411b17d1ba25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24a9611d-c416-4e9f-bfa9-411b17d1ba25" (UID: "24a9611d-c416-4e9f-bfa9-411b17d1ba25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:29.112562 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.112529    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a9611d-c416-4e9f-bfa9-411b17d1ba25-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:38:29.778640 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.778615    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"
Jul 16 22:38:29.779100 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.778652    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq" event={"ID":"24a9611d-c416-4e9f-bfa9-411b17d1ba25","Type":"ContainerDied","Data":"c4bf9c650932458bfb2e67f2b32d26a4df190f418c46922f3db133d5d886e57c"}
Jul 16 22:38:29.779100 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.778712    2555 scope.go:117] "RemoveContainer" containerID="0351e5a95577c12a4cfbe19124e0229c4c6197c4e6d5845fdc2a9f0da0efdbb7"
Jul 16 22:38:29.780681 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.780653    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerID="6206977891f7a95e4cbf806ce98321c0e4a64ada7719336c730dee557a09ffd0" exitCode=6
Jul 16 22:38:29.780804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.780705    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" event={"ID":"5a553e9b-dc46-42fb-9f92-f57d0cfbff89","Type":"ContainerDied","Data":"6206977891f7a95e4cbf806ce98321c0e4a64ada7719336c730dee557a09ffd0"}
Jul 16 22:38:29.781006 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.780976    2555 scope.go:117] "RemoveContainer" containerID="6206977891f7a95e4cbf806ce98321c0e4a64ada7719336c730dee557a09ffd0"
Jul 16 22:38:29.781187 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:29.781168    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97_odh-ai-gateway-infra(5a553e9b-dc46-42fb-9f92-f57d0cfbff89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89"
Jul 16 22:38:29.787543 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.787383    2555 scope.go:117] "RemoveContainer" containerID="23e02d8e9f123cc5b13b3f1fdea3be82467747b18d55841ca5ee9a8d393aec5b"
Jul 16 22:38:29.857687 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.857653    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"]
Jul 16 22:38:29.864085 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:29.864059    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-6cnnq"]
Jul 16 22:38:30.496021 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:30.495990    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" path="/var/lib/kubelet/pods/24a9611d-c416-4e9f-bfa9-411b17d1ba25/volumes"
Jul 16 22:38:30.787615 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:30.787537    2555 scope.go:117] "RemoveContainer" containerID="6206977891f7a95e4cbf806ce98321c0e4a64ada7719336c730dee557a09ffd0"
Jul 16 22:38:30.787992 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:30.787702    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97_odh-ai-gateway-infra(5a553e9b-dc46-42fb-9f92-f57d0cfbff89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89"
Jul 16 22:38:41.490062 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:41.490030    2555 scope.go:117] "RemoveContainer" containerID="6206977891f7a95e4cbf806ce98321c0e4a64ada7719336c730dee557a09ffd0"
Jul 16 22:38:41.839451 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:41.839410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" event={"ID":"5a553e9b-dc46-42fb-9f92-f57d0cfbff89","Type":"ContainerStarted","Data":"0af25cf4f9249c5f0db5959bad728f4dc0414419b9539bbc04438f0ad02275ed"}
Jul 16 22:38:41.858766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:41.858715    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" podStartSLOduration=14.858699497 podStartE2EDuration="14.858699497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:41.856073356 +0000 UTC m=+3077.849237524" watchObservedRunningTime="2026-07-16 22:38:41.858699497 +0000 UTC m=+3077.851863649"
Jul 16 22:38:42.845515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:42.845483    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerID="0af25cf4f9249c5f0db5959bad728f4dc0414419b9539bbc04438f0ad02275ed" exitCode=6
Jul 16 22:38:42.845982 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:42.845548    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" event={"ID":"5a553e9b-dc46-42fb-9f92-f57d0cfbff89","Type":"ContainerDied","Data":"0af25cf4f9249c5f0db5959bad728f4dc0414419b9539bbc04438f0ad02275ed"}
Jul 16 22:38:42.845982 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:42.845614    2555 scope.go:117] "RemoveContainer" containerID="6206977891f7a95e4cbf806ce98321c0e4a64ada7719336c730dee557a09ffd0"
Jul 16 22:38:42.845982 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:42.845909    2555 scope.go:117] "RemoveContainer" containerID="0af25cf4f9249c5f0db5959bad728f4dc0414419b9539bbc04438f0ad02275ed"
Jul 16 22:38:42.846157 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:38:42.846138    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97_odh-ai-gateway-infra(5a553e9b-dc46-42fb-9f92-f57d0cfbff89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89"
Jul 16 22:38:42.873975 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:42.873926    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"]
Jul 16 22:38:43.997003 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:43.996976    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"
Jul 16 22:38:44.040086 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.040046    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a553e9b-dc46-42fb-9f92-f57d0cfbff89-maas-api-service-ca\") pod \"5a553e9b-dc46-42fb-9f92-f57d0cfbff89\" (UID: \"5a553e9b-dc46-42fb-9f92-f57d0cfbff89\") "
Jul 16 22:38:44.040361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.040338    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a553e9b-dc46-42fb-9f92-f57d0cfbff89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a553e9b-dc46-42fb-9f92-f57d0cfbff89" (UID: "5a553e9b-dc46-42fb-9f92-f57d0cfbff89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:44.140637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.140546    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a553e9b-dc46-42fb-9f92-f57d0cfbff89-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:38:44.856532 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.856497    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"
Jul 16 22:38:44.856532 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.856516    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97" event={"ID":"5a553e9b-dc46-42fb-9f92-f57d0cfbff89","Type":"ContainerDied","Data":"74b3725a1feb6857f1f08454a0289b20d14a00d54d8671d84c8fddb637ceeec7"}
Jul 16 22:38:44.856796 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.856583    2555 scope.go:117] "RemoveContainer" containerID="0af25cf4f9249c5f0db5959bad728f4dc0414419b9539bbc04438f0ad02275ed"
Jul 16 22:38:44.872190 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.872165    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"]
Jul 16 22:38:44.876261 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:44.876237    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-mnh97"]
Jul 16 22:38:46.495386 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:38:46.495349    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" path="/var/lib/kubelet/pods/5a553e9b-dc46-42fb-9f92-f57d0cfbff89/volumes"
Jul 16 22:39:05.764062 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764029    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"]
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764316    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764327    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764335    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764340    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764351    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764358    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764364    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764370    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764379    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764384    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764393    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764399    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764443    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764451    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764458    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60974fa-3cda-4074-b52a-20792c0d863b" containerName="revoke-keys"
Jul 16 22:39:05.764459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764466    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:05.765075 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.764471    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:05.767385 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.767369    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"
Jul 16 22:39:05.770456 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.770432    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:39:05.770767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.770747    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:39:05.776949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.776927    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"]
Jul 16 22:39:05.827401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.827365    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c08e68-72e0-45d3-867d-d8cc205075af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb\" (UID: \"73c08e68-72e0-45d3-867d-d8cc205075af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"
Jul 16 22:39:05.928103 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.928063    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c08e68-72e0-45d3-867d-d8cc205075af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb\" (UID: \"73c08e68-72e0-45d3-867d-d8cc205075af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"
Jul 16 22:39:05.928674 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:05.928654    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c08e68-72e0-45d3-867d-d8cc205075af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb\" (UID: \"73c08e68-72e0-45d3-867d-d8cc205075af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"
Jul 16 22:39:06.078570 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.078537    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"
Jul 16 22:39:06.207217 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.207186    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"]
Jul 16 22:39:06.208131 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:39:06.208106    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c08e68_72e0_45d3_867d_d8cc205075af.slice/crio-1fb674b64560004356ab1cf133ff3dde1de87fad422126ce6cb5ef9658100a88 WatchSource:0}: Error finding container 1fb674b64560004356ab1cf133ff3dde1de87fad422126ce6cb5ef9658100a88: Status 404 returned error can't find the container with id 1fb674b64560004356ab1cf133ff3dde1de87fad422126ce6cb5ef9658100a88
Jul 16 22:39:06.405773 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.405689    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"]
Jul 16 22:39:06.406172 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.406156    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:06.406233 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.406176    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:06.406270 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.406248    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a9611d-c416-4e9f-bfa9-411b17d1ba25" containerName="revoke-keys"
Jul 16 22:39:06.406270 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.406261    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a553e9b-dc46-42fb-9f92-f57d0cfbff89" containerName="revoke-keys"
Jul 16 22:39:06.410189 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.410172    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"
Jul 16 22:39:06.416373 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.416344    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"]
Jul 16 22:39:06.532928 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.532892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a26cde76-3d2a-4312-8d90-3442bcb7a0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw\" (UID: \"a26cde76-3d2a-4312-8d90-3442bcb7a0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"
Jul 16 22:39:06.633534 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.633491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a26cde76-3d2a-4312-8d90-3442bcb7a0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw\" (UID: \"a26cde76-3d2a-4312-8d90-3442bcb7a0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"
Jul 16 22:39:06.634171 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.634154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a26cde76-3d2a-4312-8d90-3442bcb7a0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw\" (UID: \"a26cde76-3d2a-4312-8d90-3442bcb7a0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"
Jul 16 22:39:06.722645 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.722558    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"
Jul 16 22:39:06.852851 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.852654    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"]
Jul 16 22:39:06.854952 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:39:06.854919    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26cde76_3d2a_4312_8d90_3442bcb7a0fe.slice/crio-5a33ad99099dd0367d060db623b0603d7bc6ac916d6708326c59f7279ebc483d WatchSource:0}: Error finding container 5a33ad99099dd0367d060db623b0603d7bc6ac916d6708326c59f7279ebc483d: Status 404 returned error can't find the container with id 5a33ad99099dd0367d060db623b0603d7bc6ac916d6708326c59f7279ebc483d
Jul 16 22:39:06.954881 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.954849    2555 generic.go:356] "Generic (PLEG): container finished" podID="73c08e68-72e0-45d3-867d-d8cc205075af" containerID="38029ebbcf2e2fd34bb144dd32b14ecc3bed94ef99e8150f17c06d871c396860" exitCode=6
Jul 16 22:39:06.955044 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.954941    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" event={"ID":"73c08e68-72e0-45d3-867d-d8cc205075af","Type":"ContainerDied","Data":"38029ebbcf2e2fd34bb144dd32b14ecc3bed94ef99e8150f17c06d871c396860"}
Jul 16 22:39:06.955044 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.955010    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" event={"ID":"73c08e68-72e0-45d3-867d-d8cc205075af","Type":"ContainerStarted","Data":"1fb674b64560004356ab1cf133ff3dde1de87fad422126ce6cb5ef9658100a88"}
Jul 16 22:39:06.955165 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.955059    2555 scope.go:117] "RemoveContainer" containerID="38029ebbcf2e2fd34bb144dd32b14ecc3bed94ef99e8150f17c06d871c396860"
Jul 16 22:39:06.956118 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:06.956094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" event={"ID":"a26cde76-3d2a-4312-8d90-3442bcb7a0fe","Type":"ContainerStarted","Data":"5a33ad99099dd0367d060db623b0603d7bc6ac916d6708326c59f7279ebc483d"}
Jul 16 22:39:07.962454 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:07.962420    2555 generic.go:356] "Generic (PLEG): container finished" podID="73c08e68-72e0-45d3-867d-d8cc205075af" containerID="01a2e97dc4a8a7e21c2e4e1a436fc343583084bfd762402af396b48fff109093" exitCode=6
Jul 16 22:39:07.962996 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:07.962494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" event={"ID":"73c08e68-72e0-45d3-867d-d8cc205075af","Type":"ContainerDied","Data":"01a2e97dc4a8a7e21c2e4e1a436fc343583084bfd762402af396b48fff109093"}
Jul 16 22:39:07.962996 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:07.962554    2555 scope.go:117] "RemoveContainer" containerID="38029ebbcf2e2fd34bb144dd32b14ecc3bed94ef99e8150f17c06d871c396860"
Jul 16 22:39:07.962996 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:07.962729    2555 scope.go:117] "RemoveContainer" containerID="01a2e97dc4a8a7e21c2e4e1a436fc343583084bfd762402af396b48fff109093"
Jul 16 22:39:07.963194 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:07.962998    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb_odh-ai-gateway-infra(73c08e68-72e0-45d3-867d-d8cc205075af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" podUID="73c08e68-72e0-45d3-867d-d8cc205075af"
Jul 16 22:39:07.964142 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:07.964124    2555 generic.go:356] "Generic (PLEG): container finished" podID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerID="7d60084185e15358767c04d49f4ff081985a7f5d553de4f8fce98ce6f1bb77c0" exitCode=6
Jul 16 22:39:07.964250 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:07.964160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" event={"ID":"a26cde76-3d2a-4312-8d90-3442bcb7a0fe","Type":"ContainerDied","Data":"7d60084185e15358767c04d49f4ff081985a7f5d553de4f8fce98ce6f1bb77c0"}
Jul 16 22:39:07.964383 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:07.964367    2555 scope.go:117] "RemoveContainer" containerID="7d60084185e15358767c04d49f4ff081985a7f5d553de4f8fce98ce6f1bb77c0"
Jul 16 22:39:08.789332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.789248    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"]
Jul 16 22:39:08.792421 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.792402    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"
Jul 16 22:39:08.804742 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.804713    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"]
Jul 16 22:39:08.854724 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.854687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07acbd54-d3e6-4c3d-a863-097c665d46bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5\" (UID: \"07acbd54-d3e6-4c3d-a863-097c665d46bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"
Jul 16 22:39:08.956051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.956018    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07acbd54-d3e6-4c3d-a863-097c665d46bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5\" (UID: \"07acbd54-d3e6-4c3d-a863-097c665d46bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"
Jul 16 22:39:08.956613 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.956594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07acbd54-d3e6-4c3d-a863-097c665d46bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5\" (UID: \"07acbd54-d3e6-4c3d-a863-097c665d46bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"
Jul 16 22:39:08.970133 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.970112    2555 scope.go:117] "RemoveContainer" containerID="01a2e97dc4a8a7e21c2e4e1a436fc343583084bfd762402af396b48fff109093"
Jul 16 22:39:08.970515 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:08.970321    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb_odh-ai-gateway-infra(73c08e68-72e0-45d3-867d-d8cc205075af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" podUID="73c08e68-72e0-45d3-867d-d8cc205075af"
Jul 16 22:39:08.971521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.971498    2555 generic.go:356] "Generic (PLEG): container finished" podID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerID="55cf531f9638f1f404b1a92746b3d971987831d09872d6e983127f4f5b7ca8e3" exitCode=6
Jul 16 22:39:08.971631 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.971563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" event={"ID":"a26cde76-3d2a-4312-8d90-3442bcb7a0fe","Type":"ContainerDied","Data":"55cf531f9638f1f404b1a92746b3d971987831d09872d6e983127f4f5b7ca8e3"}
Jul 16 22:39:08.971631 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.971612    2555 scope.go:117] "RemoveContainer" containerID="7d60084185e15358767c04d49f4ff081985a7f5d553de4f8fce98ce6f1bb77c0"
Jul 16 22:39:08.971861 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:08.971844    2555 scope.go:117] "RemoveContainer" containerID="55cf531f9638f1f404b1a92746b3d971987831d09872d6e983127f4f5b7ca8e3"
Jul 16 22:39:08.972065 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:08.972047    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw_odh-ai-gateway-infra(a26cde76-3d2a-4312-8d90-3442bcb7a0fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe"
Jul 16 22:39:09.102531 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:09.102502    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"
Jul 16 22:39:09.239260 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:09.239237    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"]
Jul 16 22:39:09.241762 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:39:09.241735    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07acbd54_d3e6_4c3d_a863_097c665d46bf.slice/crio-836f256aaf2979e5bbaadee7bb2957164cb650888ed5bb90a2704121fe1096f8 WatchSource:0}: Error finding container 836f256aaf2979e5bbaadee7bb2957164cb650888ed5bb90a2704121fe1096f8: Status 404 returned error can't find the container with id 836f256aaf2979e5bbaadee7bb2957164cb650888ed5bb90a2704121fe1096f8
Jul 16 22:39:09.977768 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:09.977730    2555 generic.go:356] "Generic (PLEG): container finished" podID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerID="6096704adc7702856d05ac698569c872687ca1adf530dfd047778c365366bb5d" exitCode=6
Jul 16 22:39:09.978236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:09.977814    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" event={"ID":"07acbd54-d3e6-4c3d-a863-097c665d46bf","Type":"ContainerDied","Data":"6096704adc7702856d05ac698569c872687ca1adf530dfd047778c365366bb5d"}
Jul 16 22:39:09.978236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:09.977873    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" event={"ID":"07acbd54-d3e6-4c3d-a863-097c665d46bf","Type":"ContainerStarted","Data":"836f256aaf2979e5bbaadee7bb2957164cb650888ed5bb90a2704121fe1096f8"}
Jul 16 22:39:09.978236 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:09.978014    2555 scope.go:117] "RemoveContainer" containerID="6096704adc7702856d05ac698569c872687ca1adf530dfd047778c365366bb5d"
Jul 16 22:39:09.979668 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:09.979650    2555 scope.go:117] "RemoveContainer" containerID="55cf531f9638f1f404b1a92746b3d971987831d09872d6e983127f4f5b7ca8e3"
Jul 16 22:39:09.979856 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:09.979838    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw_odh-ai-gateway-infra(a26cde76-3d2a-4312-8d90-3442bcb7a0fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe"
Jul 16 22:39:10.985649 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:10.985607    2555 generic.go:356] "Generic (PLEG): container finished" podID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerID="1de68a1c9c4988b50ea95887851e5a903f868cb12d939efc00474bd9609555d9" exitCode=6
Jul 16 22:39:10.986112 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:10.985700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" event={"ID":"07acbd54-d3e6-4c3d-a863-097c665d46bf","Type":"ContainerDied","Data":"1de68a1c9c4988b50ea95887851e5a903f868cb12d939efc00474bd9609555d9"}
Jul 16 22:39:10.986112 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:10.985772    2555 scope.go:117] "RemoveContainer" containerID="6096704adc7702856d05ac698569c872687ca1adf530dfd047778c365366bb5d"
Jul 16 22:39:10.986112 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:10.985937    2555 scope.go:117] "RemoveContainer" containerID="1de68a1c9c4988b50ea95887851e5a903f868cb12d939efc00474bd9609555d9"
Jul 16 22:39:10.986234 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:10.986182    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5_odh-ai-gateway-infra(07acbd54-d3e6-4c3d-a863-097c665d46bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf"
Jul 16 22:39:11.992633 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:11.992609    2555 scope.go:117] "RemoveContainer" containerID="1de68a1c9c4988b50ea95887851e5a903f868cb12d939efc00474bd9609555d9"
Jul 16 22:39:11.993035 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:11.992795    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5_odh-ai-gateway-infra(07acbd54-d3e6-4c3d-a863-097c665d46bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf"
Jul 16 22:39:17.537276 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.537230    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"]
Jul 16 22:39:17.542571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.542549    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"
Jul 16 22:39:17.553325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.553288    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"]
Jul 16 22:39:17.635491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.635451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b92a281-5bc2-4738-8225-4e6990883bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp\" (UID: \"8b92a281-5bc2-4738-8225-4e6990883bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"
Jul 16 22:39:17.736593 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.736539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b92a281-5bc2-4738-8225-4e6990883bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp\" (UID: \"8b92a281-5bc2-4738-8225-4e6990883bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"
Jul 16 22:39:17.737401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.737377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b92a281-5bc2-4738-8225-4e6990883bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp\" (UID: \"8b92a281-5bc2-4738-8225-4e6990883bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"
Jul 16 22:39:17.853780 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.853748    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"
Jul 16 22:39:17.984715 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:17.984689    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"]
Jul 16 22:39:17.986387 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:39:17.986358    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b92a281_5bc2_4738_8225_4e6990883bf1.slice/crio-9cb9875fd21f9b2a878ecf65a89af9a3bc494306524449e59e6f9fcfd52bd078 WatchSource:0}: Error finding container 9cb9875fd21f9b2a878ecf65a89af9a3bc494306524449e59e6f9fcfd52bd078: Status 404 returned error can't find the container with id 9cb9875fd21f9b2a878ecf65a89af9a3bc494306524449e59e6f9fcfd52bd078
Jul 16 22:39:18.022669 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:18.022641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" event={"ID":"8b92a281-5bc2-4738-8225-4e6990883bf1","Type":"ContainerStarted","Data":"9cb9875fd21f9b2a878ecf65a89af9a3bc494306524449e59e6f9fcfd52bd078"}
Jul 16 22:39:19.028007 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:19.027947    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerID="73a6309fb1edbee6738520e6b0e44ec7d7f2f21cd0921c0bd0c4133f93cdc90a" exitCode=6
Jul 16 22:39:19.028406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:19.028036    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" event={"ID":"8b92a281-5bc2-4738-8225-4e6990883bf1","Type":"ContainerDied","Data":"73a6309fb1edbee6738520e6b0e44ec7d7f2f21cd0921c0bd0c4133f93cdc90a"}
Jul 16 22:39:19.028406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:19.028229    2555 scope.go:117] "RemoveContainer" containerID="73a6309fb1edbee6738520e6b0e44ec7d7f2f21cd0921c0bd0c4133f93cdc90a"
Jul 16 22:39:20.034920 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:20.034832    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerID="dbe3f752d9459ac90e539be6c96835c863b6eb3cc39aceb9846ba1ccdff1cef2" exitCode=6
Jul 16 22:39:20.034920 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:20.034905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" event={"ID":"8b92a281-5bc2-4738-8225-4e6990883bf1","Type":"ContainerDied","Data":"dbe3f752d9459ac90e539be6c96835c863b6eb3cc39aceb9846ba1ccdff1cef2"}
Jul 16 22:39:20.035405 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:20.035001    2555 scope.go:117] "RemoveContainer" containerID="73a6309fb1edbee6738520e6b0e44ec7d7f2f21cd0921c0bd0c4133f93cdc90a"
Jul 16 22:39:20.035405 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:20.035161    2555 scope.go:117] "RemoveContainer" containerID="dbe3f752d9459ac90e539be6c96835c863b6eb3cc39aceb9846ba1ccdff1cef2"
Jul 16 22:39:20.035405 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:20.035386    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp_odh-ai-gateway-infra(8b92a281-5bc2-4738-8225-4e6990883bf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1"
Jul 16 22:39:20.490519 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:20.490484    2555 scope.go:117] "RemoveContainer" containerID="01a2e97dc4a8a7e21c2e4e1a436fc343583084bfd762402af396b48fff109093"
Jul 16 22:39:21.043323 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:21.043293    2555 scope.go:117] "RemoveContainer" containerID="dbe3f752d9459ac90e539be6c96835c863b6eb3cc39aceb9846ba1ccdff1cef2"
Jul 16 22:39:21.043754 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:21.043527    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp_odh-ai-gateway-infra(8b92a281-5bc2-4738-8225-4e6990883bf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1"
Jul 16 22:39:21.045075 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:21.045048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" event={"ID":"73c08e68-72e0-45d3-867d-d8cc205075af","Type":"ContainerStarted","Data":"becabd0c8a44e8d0cea472427f5da53aa27d5e30cb23fd44c3529d49ab7ad1bd"}
Jul 16 22:39:21.085517 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:21.085463    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" podStartSLOduration=16.085445625 podStartE2EDuration="16.085445625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:21.085441233 +0000 UTC m=+3117.078605385" watchObservedRunningTime="2026-07-16 22:39:21.085445625 +0000 UTC m=+3117.078609822"
Jul 16 22:39:22.052064 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:22.052029    2555 generic.go:356] "Generic (PLEG): container finished" podID="73c08e68-72e0-45d3-867d-d8cc205075af" containerID="becabd0c8a44e8d0cea472427f5da53aa27d5e30cb23fd44c3529d49ab7ad1bd" exitCode=6
Jul 16 22:39:22.052491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:22.052102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" event={"ID":"73c08e68-72e0-45d3-867d-d8cc205075af","Type":"ContainerDied","Data":"becabd0c8a44e8d0cea472427f5da53aa27d5e30cb23fd44c3529d49ab7ad1bd"}
Jul 16 22:39:22.052491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:22.052159    2555 scope.go:117] "RemoveContainer" containerID="01a2e97dc4a8a7e21c2e4e1a436fc343583084bfd762402af396b48fff109093"
Jul 16 22:39:22.052491 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:22.052455    2555 scope.go:117] "RemoveContainer" containerID="becabd0c8a44e8d0cea472427f5da53aa27d5e30cb23fd44c3529d49ab7ad1bd"
Jul 16 22:39:22.052769 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:22.052744    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb_odh-ai-gateway-infra(73c08e68-72e0-45d3-867d-d8cc205075af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" podUID="73c08e68-72e0-45d3-867d-d8cc205075af"
Jul 16 22:39:22.095950 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:22.095916    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"]
Jul 16 22:39:23.189210 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:23.189186    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"
Jul 16 22:39:23.288696 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:23.288657    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c08e68-72e0-45d3-867d-d8cc205075af-maas-api-service-ca\") pod \"73c08e68-72e0-45d3-867d-d8cc205075af\" (UID: \"73c08e68-72e0-45d3-867d-d8cc205075af\") "
Jul 16 22:39:23.289055 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:23.289030    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c08e68-72e0-45d3-867d-d8cc205075af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73c08e68-72e0-45d3-867d-d8cc205075af" (UID: "73c08e68-72e0-45d3-867d-d8cc205075af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:23.390071 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:23.390028    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c08e68-72e0-45d3-867d-d8cc205075af-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.067930 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:24.067897    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"
Jul 16 22:39:24.068128 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:24.067933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb" event={"ID":"73c08e68-72e0-45d3-867d-d8cc205075af","Type":"ContainerDied","Data":"1fb674b64560004356ab1cf133ff3dde1de87fad422126ce6cb5ef9658100a88"}
Jul 16 22:39:24.068128 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:24.068064    2555 scope.go:117] "RemoveContainer" containerID="becabd0c8a44e8d0cea472427f5da53aa27d5e30cb23fd44c3529d49ab7ad1bd"
Jul 16 22:39:24.090546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:24.090521    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"]
Jul 16 22:39:24.094346 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:24.094320    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-8n2vb"]
Jul 16 22:39:24.491633 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:24.491598    2555 scope.go:117] "RemoveContainer" containerID="55cf531f9638f1f404b1a92746b3d971987831d09872d6e983127f4f5b7ca8e3"
Jul 16 22:39:24.494921 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:24.494895    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" path="/var/lib/kubelet/pods/73c08e68-72e0-45d3-867d-d8cc205075af/volumes"
Jul 16 22:39:25.074999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:25.074948    2555 generic.go:356] "Generic (PLEG): container finished" podID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerID="b3be291278fd7c18458b536e9f896e2144221263b4559b13f739a100dc408bcb" exitCode=6
Jul 16 22:39:25.074999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:25.074987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" event={"ID":"a26cde76-3d2a-4312-8d90-3442bcb7a0fe","Type":"ContainerDied","Data":"b3be291278fd7c18458b536e9f896e2144221263b4559b13f739a100dc408bcb"}
Jul 16 22:39:25.075186 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:25.075055    2555 scope.go:117] "RemoveContainer" containerID="55cf531f9638f1f404b1a92746b3d971987831d09872d6e983127f4f5b7ca8e3"
Jul 16 22:39:25.075383 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:25.075363    2555 scope.go:117] "RemoveContainer" containerID="b3be291278fd7c18458b536e9f896e2144221263b4559b13f739a100dc408bcb"
Jul 16 22:39:25.075600 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:25.075580    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw_odh-ai-gateway-infra(a26cde76-3d2a-4312-8d90-3442bcb7a0fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe"
Jul 16 22:39:26.102727 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:26.102691    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"]
Jul 16 22:39:26.236312 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:26.236287    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"
Jul 16 22:39:26.414182 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:26.414084    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a26cde76-3d2a-4312-8d90-3442bcb7a0fe-maas-api-service-ca\") pod \"a26cde76-3d2a-4312-8d90-3442bcb7a0fe\" (UID: \"a26cde76-3d2a-4312-8d90-3442bcb7a0fe\") "
Jul 16 22:39:26.414424 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:26.414401    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26cde76-3d2a-4312-8d90-3442bcb7a0fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a26cde76-3d2a-4312-8d90-3442bcb7a0fe" (UID: "a26cde76-3d2a-4312-8d90-3442bcb7a0fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:26.515535 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:26.515504    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a26cde76-3d2a-4312-8d90-3442bcb7a0fe-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:39:27.089444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:27.089412    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"
Jul 16 22:39:27.089631 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:27.089445    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw" event={"ID":"a26cde76-3d2a-4312-8d90-3442bcb7a0fe","Type":"ContainerDied","Data":"5a33ad99099dd0367d060db623b0603d7bc6ac916d6708326c59f7279ebc483d"}
Jul 16 22:39:27.089631 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:27.089527    2555 scope.go:117] "RemoveContainer" containerID="b3be291278fd7c18458b536e9f896e2144221263b4559b13f739a100dc408bcb"
Jul 16 22:39:27.112661 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:27.112627    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"]
Jul 16 22:39:27.117174 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:27.117145    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-9lkqw"]
Jul 16 22:39:27.489978 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:27.489874    2555 scope.go:117] "RemoveContainer" containerID="1de68a1c9c4988b50ea95887851e5a903f868cb12d939efc00474bd9609555d9"
Jul 16 22:39:28.097442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:28.097407    2555 generic.go:356] "Generic (PLEG): container finished" podID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerID="8796895dc074f950590c2bb889f6c3e4a9d2a7b58a4bce3189bffaae9774d619" exitCode=6
Jul 16 22:39:28.097442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:28.097445    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" event={"ID":"07acbd54-d3e6-4c3d-a863-097c665d46bf","Type":"ContainerDied","Data":"8796895dc074f950590c2bb889f6c3e4a9d2a7b58a4bce3189bffaae9774d619"}
Jul 16 22:39:28.097697 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:28.097493    2555 scope.go:117] "RemoveContainer" containerID="1de68a1c9c4988b50ea95887851e5a903f868cb12d939efc00474bd9609555d9"
Jul 16 22:39:28.097836 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:28.097820    2555 scope.go:117] "RemoveContainer" containerID="8796895dc074f950590c2bb889f6c3e4a9d2a7b58a4bce3189bffaae9774d619"
Jul 16 22:39:28.098070 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:28.098044    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5_odh-ai-gateway-infra(07acbd54-d3e6-4c3d-a863-097c665d46bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf"
Jul 16 22:39:28.496798 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:28.496764    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" path="/var/lib/kubelet/pods/a26cde76-3d2a-4312-8d90-3442bcb7a0fe/volumes"
Jul 16 22:39:29.125209 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:29.125178    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"]
Jul 16 22:39:29.265034 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:29.265010    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"
Jul 16 22:39:29.440728 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:29.440632    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07acbd54-d3e6-4c3d-a863-097c665d46bf-maas-api-service-ca\") pod \"07acbd54-d3e6-4c3d-a863-097c665d46bf\" (UID: \"07acbd54-d3e6-4c3d-a863-097c665d46bf\") "
Jul 16 22:39:29.441076 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:29.441051    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07acbd54-d3e6-4c3d-a863-097c665d46bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07acbd54-d3e6-4c3d-a863-097c665d46bf" (UID: "07acbd54-d3e6-4c3d-a863-097c665d46bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:29.542135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:29.542101    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07acbd54-d3e6-4c3d-a863-097c665d46bf-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:39:30.109571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:30.109542    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"
Jul 16 22:39:30.109571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:30.109566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5" event={"ID":"07acbd54-d3e6-4c3d-a863-097c665d46bf","Type":"ContainerDied","Data":"836f256aaf2979e5bbaadee7bb2957164cb650888ed5bb90a2704121fe1096f8"}
Jul 16 22:39:30.109828 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:30.109622    2555 scope.go:117] "RemoveContainer" containerID="8796895dc074f950590c2bb889f6c3e4a9d2a7b58a4bce3189bffaae9774d619"
Jul 16 22:39:30.141384 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:30.141353    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"]
Jul 16 22:39:30.152150 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:30.152122    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-wv5n5"]
Jul 16 22:39:30.500755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:30.500678    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" path="/var/lib/kubelet/pods/07acbd54-d3e6-4c3d-a863-097c665d46bf/volumes"
Jul 16 22:39:35.490394 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:35.490363    2555 scope.go:117] "RemoveContainer" containerID="dbe3f752d9459ac90e539be6c96835c863b6eb3cc39aceb9846ba1ccdff1cef2"
Jul 16 22:39:36.137400 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:36.137366    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerID="4b73c423ed1e1e7ba012de96cb5d27b4edb2743d0d3c02f5b777ce028d02064a" exitCode=6
Jul 16 22:39:36.137543 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:36.137439    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" event={"ID":"8b92a281-5bc2-4738-8225-4e6990883bf1","Type":"ContainerDied","Data":"4b73c423ed1e1e7ba012de96cb5d27b4edb2743d0d3c02f5b777ce028d02064a"}
Jul 16 22:39:36.137543 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:36.137509    2555 scope.go:117] "RemoveContainer" containerID="dbe3f752d9459ac90e539be6c96835c863b6eb3cc39aceb9846ba1ccdff1cef2"
Jul 16 22:39:36.137891 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:36.137872    2555 scope.go:117] "RemoveContainer" containerID="4b73c423ed1e1e7ba012de96cb5d27b4edb2743d0d3c02f5b777ce028d02064a"
Jul 16 22:39:36.138088 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:36.138068    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp_odh-ai-gateway-infra(8b92a281-5bc2-4738-8225-4e6990883bf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1"
Jul 16 22:39:37.163633 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:37.163599    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"]
Jul 16 22:39:37.290117 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:37.290086    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"
Jul 16 22:39:37.308876 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:37.308849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b92a281-5bc2-4738-8225-4e6990883bf1-maas-api-service-ca\") pod \"8b92a281-5bc2-4738-8225-4e6990883bf1\" (UID: \"8b92a281-5bc2-4738-8225-4e6990883bf1\") "
Jul 16 22:39:37.309231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:37.309208    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b92a281-5bc2-4738-8225-4e6990883bf1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b92a281-5bc2-4738-8225-4e6990883bf1" (UID: "8b92a281-5bc2-4738-8225-4e6990883bf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:37.410252 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:37.410225    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b92a281-5bc2-4738-8225-4e6990883bf1-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:39:38.150186 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:38.150155    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"
Jul 16 22:39:38.150366 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:38.150184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp" event={"ID":"8b92a281-5bc2-4738-8225-4e6990883bf1","Type":"ContainerDied","Data":"9cb9875fd21f9b2a878ecf65a89af9a3bc494306524449e59e6f9fcfd52bd078"}
Jul 16 22:39:38.150366 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:38.150243    2555 scope.go:117] "RemoveContainer" containerID="4b73c423ed1e1e7ba012de96cb5d27b4edb2743d0d3c02f5b777ce028d02064a"
Jul 16 22:39:38.175578 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:38.175550    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"]
Jul 16 22:39:38.184197 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:38.184172    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-blkqp"]
Jul 16 22:39:38.495251 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:38.495174    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" path="/var/lib/kubelet/pods/8b92a281-5bc2-4738-8225-4e6990883bf1/volumes"
Jul 16 22:39:55.813968 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.813918    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"]
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814235    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814247    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814256    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814261    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814268    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814274    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814283    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814289    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814298    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814304    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814312    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814318    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814354    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814359    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814365    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814370    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814376    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:55.814408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814381    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814426    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814433    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814445    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814451    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814458    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814463    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814469    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814475    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:55.815026 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.814481    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:55.817325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.817310    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"
Jul 16 22:39:55.820035 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.820013    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:39:55.820159 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.820086    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:39:55.826538 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.826510    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"]
Jul 16 22:39:55.875328 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.875288    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe051b75-d462-43c0-bf53-cab2380354a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq\" (UID: \"fe051b75-d462-43c0-bf53-cab2380354a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"
Jul 16 22:39:55.976327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.976285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe051b75-d462-43c0-bf53-cab2380354a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq\" (UID: \"fe051b75-d462-43c0-bf53-cab2380354a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"
Jul 16 22:39:55.976903 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:55.976883    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe051b75-d462-43c0-bf53-cab2380354a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq\" (UID: \"fe051b75-d462-43c0-bf53-cab2380354a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"
Jul 16 22:39:56.128520 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.128478    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"
Jul 16 22:39:56.257459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.257434    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"]
Jul 16 22:39:56.259592 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:39:56.259558    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe051b75_d462_43c0_bf53_cab2380354a1.slice/crio-71cb8263e195a1188c44805a4cdda6019710c43a7fe180e59418fb52c5d563f9 WatchSource:0}: Error finding container 71cb8263e195a1188c44805a4cdda6019710c43a7fe180e59418fb52c5d563f9: Status 404 returned error can't find the container with id 71cb8263e195a1188c44805a4cdda6019710c43a7fe180e59418fb52c5d563f9
Jul 16 22:39:56.635417 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635384    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rx96x"]
Jul 16 22:39:56.635724 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635711    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:56.635773 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635726    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:56.635773 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635737    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:56.635773 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635742    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07acbd54-d3e6-4c3d-a863-097c665d46bf" containerName="revoke-keys"
Jul 16 22:39:56.635773 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635755    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:56.635773 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635762    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:56.635924 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635816    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c08e68-72e0-45d3-867d-d8cc205075af" containerName="revoke-keys"
Jul 16 22:39:56.635924 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635825    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26cde76-3d2a-4312-8d90-3442bcb7a0fe" containerName="revoke-keys"
Jul 16 22:39:56.635924 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.635831    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b92a281-5bc2-4738-8225-4e6990883bf1" containerName="revoke-keys"
Jul 16 22:39:56.638950 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.638931    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:39:56.642591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.642572    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-svkfl\""
Jul 16 22:39:56.647628 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.647607    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rx96x"]
Jul 16 22:39:56.783694 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.783657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdg4p\" (UniqueName: \"kubernetes.io/projected/3844dcaa-28d0-4d4c-9611-f83ca01cef44-kube-api-access-bdg4p\") pod \"kuadrant-operator-catalog-rx96x\" (UID: \"3844dcaa-28d0-4d4c-9611-f83ca01cef44\") " pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:39:56.885005 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.884940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdg4p\" (UniqueName: \"kubernetes.io/projected/3844dcaa-28d0-4d4c-9611-f83ca01cef44-kube-api-access-bdg4p\") pod \"kuadrant-operator-catalog-rx96x\" (UID: \"3844dcaa-28d0-4d4c-9611-f83ca01cef44\") " pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:39:56.894043 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.894025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdg4p\" (UniqueName: \"kubernetes.io/projected/3844dcaa-28d0-4d4c-9611-f83ca01cef44-kube-api-access-bdg4p\") pod \"kuadrant-operator-catalog-rx96x\" (UID: \"3844dcaa-28d0-4d4c-9611-f83ca01cef44\") " pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:39:56.949092 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:56.949062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:39:57.080314 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:57.080293    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rx96x"]
Jul 16 22:39:57.082192 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:39:57.082166    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3844dcaa_28d0_4d4c_9611_f83ca01cef44.slice/crio-09d72f6adcbae45acc07ed32631ab47faa0c0e660750a324403b71b6e1f63228 WatchSource:0}: Error finding container 09d72f6adcbae45acc07ed32631ab47faa0c0e660750a324403b71b6e1f63228: Status 404 returned error can't find the container with id 09d72f6adcbae45acc07ed32631ab47faa0c0e660750a324403b71b6e1f63228
Jul 16 22:39:57.083549 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:57.083533    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:39:57.241006 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:57.240896    2555 generic.go:356] "Generic (PLEG): container finished" podID="fe051b75-d462-43c0-bf53-cab2380354a1" containerID="162396f543c1c22ffa1c291d607941c8480897e61078e688c98340f2f199d3e2" exitCode=6
Jul 16 22:39:57.241006 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:57.240995    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" event={"ID":"fe051b75-d462-43c0-bf53-cab2380354a1","Type":"ContainerDied","Data":"162396f543c1c22ffa1c291d607941c8480897e61078e688c98340f2f199d3e2"}
Jul 16 22:39:57.241249 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:57.241048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" event={"ID":"fe051b75-d462-43c0-bf53-cab2380354a1","Type":"ContainerStarted","Data":"71cb8263e195a1188c44805a4cdda6019710c43a7fe180e59418fb52c5d563f9"}
Jul 16 22:39:57.241297 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:57.241260    2555 scope.go:117] "RemoveContainer" containerID="162396f543c1c22ffa1c291d607941c8480897e61078e688c98340f2f199d3e2"
Jul 16 22:39:57.242441 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:57.242417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rx96x" event={"ID":"3844dcaa-28d0-4d4c-9611-f83ca01cef44","Type":"ContainerStarted","Data":"09d72f6adcbae45acc07ed32631ab47faa0c0e660750a324403b71b6e1f63228"}
Jul 16 22:39:58.249442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:58.249353    2555 generic.go:356] "Generic (PLEG): container finished" podID="fe051b75-d462-43c0-bf53-cab2380354a1" containerID="49f024514ce5e7f210c7e381d5ac602070ff0aff0c6d6b1fb25ed2b5cf5d715b" exitCode=6
Jul 16 22:39:58.249442 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:58.249409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" event={"ID":"fe051b75-d462-43c0-bf53-cab2380354a1","Type":"ContainerDied","Data":"49f024514ce5e7f210c7e381d5ac602070ff0aff0c6d6b1fb25ed2b5cf5d715b"}
Jul 16 22:39:58.249947 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:58.249483    2555 scope.go:117] "RemoveContainer" containerID="162396f543c1c22ffa1c291d607941c8480897e61078e688c98340f2f199d3e2"
Jul 16 22:39:58.249947 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:58.249658    2555 scope.go:117] "RemoveContainer" containerID="49f024514ce5e7f210c7e381d5ac602070ff0aff0c6d6b1fb25ed2b5cf5d715b"
Jul 16 22:39:58.249947 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:58.249913    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq_odh-ai-gateway-infra(fe051b75-d462-43c0-bf53-cab2380354a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" podUID="fe051b75-d462-43c0-bf53-cab2380354a1"
Jul 16 22:39:59.255327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:59.255291    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rx96x" event={"ID":"3844dcaa-28d0-4d4c-9611-f83ca01cef44","Type":"ContainerStarted","Data":"0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f"}
Jul 16 22:39:59.257025 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:59.257002    2555 scope.go:117] "RemoveContainer" containerID="49f024514ce5e7f210c7e381d5ac602070ff0aff0c6d6b1fb25ed2b5cf5d715b"
Jul 16 22:39:59.257179 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:39:59.257161    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq_odh-ai-gateway-infra(fe051b75-d462-43c0-bf53-cab2380354a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" podUID="fe051b75-d462-43c0-bf53-cab2380354a1"
Jul 16 22:39:59.272204 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:39:59.272160    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rx96x" podStartSLOduration=1.728265841 podStartE2EDuration="3.272142371s" totalImagesPullingTime="1.54387653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:59.271164624 +0000 UTC m=+3155.264328772" watchObservedRunningTime="2026-07-16 22:39:59.272142371 +0000 UTC m=+3155.265306525"
Jul 16 22:40:06.456013 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.455973    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"]
Jul 16 22:40:06.459528 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.459505    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"
Jul 16 22:40:06.468314 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.468272    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"]
Jul 16 22:40:06.571073 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.571033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d116693-c892-4fbb-bce7-77989aac5855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th\" (UID: \"5d116693-c892-4fbb-bce7-77989aac5855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"
Jul 16 22:40:06.672335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.672302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d116693-c892-4fbb-bce7-77989aac5855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th\" (UID: \"5d116693-c892-4fbb-bce7-77989aac5855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"
Jul 16 22:40:06.672931 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.672904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d116693-c892-4fbb-bce7-77989aac5855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th\" (UID: \"5d116693-c892-4fbb-bce7-77989aac5855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"
Jul 16 22:40:06.771147 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.771062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"
Jul 16 22:40:06.902196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.902171    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"]
Jul 16 22:40:06.904855 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:40:06.904825    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d116693_c892_4fbb_bce7_77989aac5855.slice/crio-fd68d32d9fa1925be9386cebb1ff09e61e66e8564b6889f5b28dbb1582f3e85b WatchSource:0}: Error finding container fd68d32d9fa1925be9386cebb1ff09e61e66e8564b6889f5b28dbb1582f3e85b: Status 404 returned error can't find the container with id fd68d32d9fa1925be9386cebb1ff09e61e66e8564b6889f5b28dbb1582f3e85b
Jul 16 22:40:06.949804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.949778    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:40:06.949951 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.949832    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:40:06.972196 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:06.972170    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:40:07.295413 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:07.295376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" event={"ID":"5d116693-c892-4fbb-bce7-77989aac5855","Type":"ContainerStarted","Data":"6463be6029679915e398f0597eb81cf70e54f4a951951094c2f0e70776f60268"}
Jul 16 22:40:07.295593 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:07.295430    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" event={"ID":"5d116693-c892-4fbb-bce7-77989aac5855","Type":"ContainerStarted","Data":"fd68d32d9fa1925be9386cebb1ff09e61e66e8564b6889f5b28dbb1582f3e85b"}
Jul 16 22:40:07.318020 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:07.317951    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" podStartSLOduration=1.317931449 podStartE2EDuration="1.317931449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:07.31638227 +0000 UTC m=+3163.309546423" watchObservedRunningTime="2026-07-16 22:40:07.317931449 +0000 UTC m=+3163.311095603"
Jul 16 22:40:07.318181 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:07.318027    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:40:07.359932 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:07.359895    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rx96x"]
Jul 16 22:40:08.301098 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:08.301057    2555 generic.go:356] "Generic (PLEG): container finished" podID="5d116693-c892-4fbb-bce7-77989aac5855" containerID="6463be6029679915e398f0597eb81cf70e54f4a951951094c2f0e70776f60268" exitCode=6
Jul 16 22:40:08.301546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:08.301150    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" event={"ID":"5d116693-c892-4fbb-bce7-77989aac5855","Type":"ContainerDied","Data":"6463be6029679915e398f0597eb81cf70e54f4a951951094c2f0e70776f60268"}
Jul 16 22:40:08.301546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:08.301325    2555 scope.go:117] "RemoveContainer" containerID="6463be6029679915e398f0597eb81cf70e54f4a951951094c2f0e70776f60268"
Jul 16 22:40:08.854239 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:08.854205    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"]
Jul 16 22:40:08.857360 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:08.857343    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"
Jul 16 22:40:08.866264 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:08.866228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"]
Jul 16 22:40:08.994673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:08.994634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7ac350-405c-4667-961e-f2213f5b96fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9\" (UID: \"3c7ac350-405c-4667-961e-f2213f5b96fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"
Jul 16 22:40:09.095527 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.095470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7ac350-405c-4667-961e-f2213f5b96fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9\" (UID: \"3c7ac350-405c-4667-961e-f2213f5b96fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"
Jul 16 22:40:09.096153 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.096130    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7ac350-405c-4667-961e-f2213f5b96fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9\" (UID: \"3c7ac350-405c-4667-961e-f2213f5b96fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"
Jul 16 22:40:09.168146 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.168066    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"
Jul 16 22:40:09.305553 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.305295    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"]
Jul 16 22:40:09.314360 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.314332    2555 generic.go:356] "Generic (PLEG): container finished" podID="5d116693-c892-4fbb-bce7-77989aac5855" containerID="3603e7b0dcfed7baff00addcfd077a66bd794909c82e6ee01a6e649e1a8a9592" exitCode=6
Jul 16 22:40:09.314596 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.314383    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" event={"ID":"5d116693-c892-4fbb-bce7-77989aac5855","Type":"ContainerDied","Data":"3603e7b0dcfed7baff00addcfd077a66bd794909c82e6ee01a6e649e1a8a9592"}
Jul 16 22:40:09.314747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.314624    2555 scope.go:117] "RemoveContainer" containerID="6463be6029679915e398f0597eb81cf70e54f4a951951094c2f0e70776f60268"
Jul 16 22:40:09.314943 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.314923    2555 scope.go:117] "RemoveContainer" containerID="3603e7b0dcfed7baff00addcfd077a66bd794909c82e6ee01a6e649e1a8a9592"
Jul 16 22:40:09.315150 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:09.315110    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th_odh-ai-gateway-infra(5d116693-c892-4fbb-bce7-77989aac5855)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" podUID="5d116693-c892-4fbb-bce7-77989aac5855"
Jul 16 22:40:09.317888 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.317865    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-rx96x" podUID="3844dcaa-28d0-4d4c-9611-f83ca01cef44" containerName="registry-server" containerID="cri-o://0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f" gracePeriod=2
Jul 16 22:40:09.569331 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.569309    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:40:09.700564 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.700482    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdg4p\" (UniqueName: \"kubernetes.io/projected/3844dcaa-28d0-4d4c-9611-f83ca01cef44-kube-api-access-bdg4p\") pod \"3844dcaa-28d0-4d4c-9611-f83ca01cef44\" (UID: \"3844dcaa-28d0-4d4c-9611-f83ca01cef44\") "
Jul 16 22:40:09.702485 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.702460    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3844dcaa-28d0-4d4c-9611-f83ca01cef44-kube-api-access-bdg4p" (OuterVolumeSpecName: "kube-api-access-bdg4p") pod "3844dcaa-28d0-4d4c-9611-f83ca01cef44" (UID: "3844dcaa-28d0-4d4c-9611-f83ca01cef44"). InnerVolumeSpecName "kube-api-access-bdg4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:09.801859 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:09.801816    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdg4p\" (UniqueName: \"kubernetes.io/projected/3844dcaa-28d0-4d4c-9611-f83ca01cef44-kube-api-access-bdg4p\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:40:10.320574 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.320481    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerID="4ad83cb1f222e7b96997b4803142dcdfff69f0e8e97f56615f3e70964958e9ad" exitCode=6
Jul 16 22:40:10.321054 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.320566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" event={"ID":"3c7ac350-405c-4667-961e-f2213f5b96fa","Type":"ContainerDied","Data":"4ad83cb1f222e7b96997b4803142dcdfff69f0e8e97f56615f3e70964958e9ad"}
Jul 16 22:40:10.321054 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.320623    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" event={"ID":"3c7ac350-405c-4667-961e-f2213f5b96fa","Type":"ContainerStarted","Data":"e86c85e155c97072c02d9e7b0ce5f7355f0a1abb83b78c5f93b5a62d5689a955"}
Jul 16 22:40:10.321054 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.320836    2555 scope.go:117] "RemoveContainer" containerID="4ad83cb1f222e7b96997b4803142dcdfff69f0e8e97f56615f3e70964958e9ad"
Jul 16 22:40:10.322334 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.322316    2555 scope.go:117] "RemoveContainer" containerID="3603e7b0dcfed7baff00addcfd077a66bd794909c82e6ee01a6e649e1a8a9592"
Jul 16 22:40:10.322507 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:10.322487    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th_odh-ai-gateway-infra(5d116693-c892-4fbb-bce7-77989aac5855)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" podUID="5d116693-c892-4fbb-bce7-77989aac5855"
Jul 16 22:40:10.323387 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.323364    2555 generic.go:356] "Generic (PLEG): container finished" podID="3844dcaa-28d0-4d4c-9611-f83ca01cef44" containerID="0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f" exitCode=0
Jul 16 22:40:10.323501 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.323394    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rx96x"
Jul 16 22:40:10.323501 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.323425    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rx96x" event={"ID":"3844dcaa-28d0-4d4c-9611-f83ca01cef44","Type":"ContainerDied","Data":"0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f"}
Jul 16 22:40:10.323501 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.323469    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rx96x" event={"ID":"3844dcaa-28d0-4d4c-9611-f83ca01cef44","Type":"ContainerDied","Data":"09d72f6adcbae45acc07ed32631ab47faa0c0e660750a324403b71b6e1f63228"}
Jul 16 22:40:10.323501 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.323494    2555 scope.go:117] "RemoveContainer" containerID="0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f"
Jul 16 22:40:10.333161 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.333144    2555 scope.go:117] "RemoveContainer" containerID="0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f"
Jul 16 22:40:10.333417 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:10.333396    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f\": container with ID starting with 0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f not found: ID does not exist" containerID="0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f"
Jul 16 22:40:10.333471 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.333425    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f"} err="failed to get container status \"0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f\": rpc error: code = NotFound desc = could not find container \"0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f\": container with ID starting with 0a540d8d49fedbac78036a38deeb8da21d8cf799337ab551ace674d6d375864f not found: ID does not exist"
Jul 16 22:40:10.415738 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.415705    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rx96x"]
Jul 16 22:40:10.424600 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.424576    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rx96x"]
Jul 16 22:40:10.494685 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:10.494654    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3844dcaa-28d0-4d4c-9611-f83ca01cef44" path="/var/lib/kubelet/pods/3844dcaa-28d0-4d4c-9611-f83ca01cef44/volumes"
Jul 16 22:40:11.329040 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:11.329004    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerID="775b8ebac07314b870b86d28875a37d77880972d262cb23614f69174ff3db4ec" exitCode=6
Jul 16 22:40:11.329496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:11.329082    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" event={"ID":"3c7ac350-405c-4667-961e-f2213f5b96fa","Type":"ContainerDied","Data":"775b8ebac07314b870b86d28875a37d77880972d262cb23614f69174ff3db4ec"}
Jul 16 22:40:11.329496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:11.329142    2555 scope.go:117] "RemoveContainer" containerID="4ad83cb1f222e7b96997b4803142dcdfff69f0e8e97f56615f3e70964958e9ad"
Jul 16 22:40:11.329496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:11.329356    2555 scope.go:117] "RemoveContainer" containerID="775b8ebac07314b870b86d28875a37d77880972d262cb23614f69174ff3db4ec"
Jul 16 22:40:11.329658 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:11.329554    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9_odh-ai-gateway-infra(3c7ac350-405c-4667-961e-f2213f5b96fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa"
Jul 16 22:40:12.336725 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:12.336699    2555 scope.go:117] "RemoveContainer" containerID="775b8ebac07314b870b86d28875a37d77880972d262cb23614f69174ff3db4ec"
Jul 16 22:40:12.337149 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:12.336877    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9_odh-ai-gateway-infra(3c7ac350-405c-4667-961e-f2213f5b96fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa"
Jul 16 22:40:12.490303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:12.490274    2555 scope.go:117] "RemoveContainer" containerID="49f024514ce5e7f210c7e381d5ac602070ff0aff0c6d6b1fb25ed2b5cf5d715b"
Jul 16 22:40:13.343672 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:13.343631    2555 generic.go:356] "Generic (PLEG): container finished" podID="fe051b75-d462-43c0-bf53-cab2380354a1" containerID="50ebdd809a7f8f9a46bd6356df92931823b8d89fb12b04c775b80f3bc7cd10ee" exitCode=6
Jul 16 22:40:13.344087 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:13.343701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" event={"ID":"fe051b75-d462-43c0-bf53-cab2380354a1","Type":"ContainerDied","Data":"50ebdd809a7f8f9a46bd6356df92931823b8d89fb12b04c775b80f3bc7cd10ee"}
Jul 16 22:40:13.344087 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:13.343766    2555 scope.go:117] "RemoveContainer" containerID="49f024514ce5e7f210c7e381d5ac602070ff0aff0c6d6b1fb25ed2b5cf5d715b"
Jul 16 22:40:13.344197 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:13.344093    2555 scope.go:117] "RemoveContainer" containerID="50ebdd809a7f8f9a46bd6356df92931823b8d89fb12b04c775b80f3bc7cd10ee"
Jul 16 22:40:13.344337 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:13.344314    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq_odh-ai-gateway-infra(fe051b75-d462-43c0-bf53-cab2380354a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" podUID="fe051b75-d462-43c0-bf53-cab2380354a1"
Jul 16 22:40:14.372209 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:14.372171    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"]
Jul 16 22:40:14.499818 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:14.499785    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"
Jul 16 22:40:14.642388 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:14.642288    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe051b75-d462-43c0-bf53-cab2380354a1-maas-api-service-ca\") pod \"fe051b75-d462-43c0-bf53-cab2380354a1\" (UID: \"fe051b75-d462-43c0-bf53-cab2380354a1\") "
Jul 16 22:40:14.642722 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:14.642694    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe051b75-d462-43c0-bf53-cab2380354a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe051b75-d462-43c0-bf53-cab2380354a1" (UID: "fe051b75-d462-43c0-bf53-cab2380354a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:14.642872 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:14.642837    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe051b75-d462-43c0-bf53-cab2380354a1-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:40:15.356496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:15.356466    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"
Jul 16 22:40:15.356669 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:15.356473    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq" event={"ID":"fe051b75-d462-43c0-bf53-cab2380354a1","Type":"ContainerDied","Data":"71cb8263e195a1188c44805a4cdda6019710c43a7fe180e59418fb52c5d563f9"}
Jul 16 22:40:15.356669 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:15.356605    2555 scope.go:117] "RemoveContainer" containerID="50ebdd809a7f8f9a46bd6356df92931823b8d89fb12b04c775b80f3bc7cd10ee"
Jul 16 22:40:15.378898 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:15.378872    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"]
Jul 16 22:40:15.384347 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:15.384326    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-zhzzq"]
Jul 16 22:40:16.494514 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:16.494475    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" path="/var/lib/kubelet/pods/fe051b75-d462-43c0-bf53-cab2380354a1/volumes"
Jul 16 22:40:17.582455 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582422    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"]
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582778    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582791    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582806    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582813    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582824    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582829    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582838    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3844dcaa-28d0-4d4c-9611-f83ca01cef44" containerName="registry-server"
Jul 16 22:40:17.582857 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582844    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3844dcaa-28d0-4d4c-9611-f83ca01cef44" containerName="registry-server"
Jul 16 22:40:17.583164 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582899    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.583164 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582907    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:17.583164 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.582914    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3844dcaa-28d0-4d4c-9611-f83ca01cef44" containerName="registry-server"
Jul 16 22:40:17.587330 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.587314    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"
Jul 16 22:40:17.596635 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.596614    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"]
Jul 16 22:40:17.769589 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.769548    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b352ec86-54b0-41d7-b839-0266d4fe0b87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2\" (UID: \"b352ec86-54b0-41d7-b839-0266d4fe0b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"
Jul 16 22:40:17.870802 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.870717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b352ec86-54b0-41d7-b839-0266d4fe0b87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2\" (UID: \"b352ec86-54b0-41d7-b839-0266d4fe0b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"
Jul 16 22:40:17.871357 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.871337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b352ec86-54b0-41d7-b839-0266d4fe0b87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2\" (UID: \"b352ec86-54b0-41d7-b839-0266d4fe0b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"
Jul 16 22:40:17.899741 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:17.899703    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"
Jul 16 22:40:18.039208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:18.033228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"]
Jul 16 22:40:18.040744 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:40:18.040710    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb352ec86_54b0_41d7_b839_0266d4fe0b87.slice/crio-8af1dcbdf8d20b1c12dbb93e0aaf4a69c7b5dabe74f21bfdd427589c4f41aecf WatchSource:0}: Error finding container 8af1dcbdf8d20b1c12dbb93e0aaf4a69c7b5dabe74f21bfdd427589c4f41aecf: Status 404 returned error can't find the container with id 8af1dcbdf8d20b1c12dbb93e0aaf4a69c7b5dabe74f21bfdd427589c4f41aecf
Jul 16 22:40:18.374162 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:18.374125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" event={"ID":"b352ec86-54b0-41d7-b839-0266d4fe0b87","Type":"ContainerStarted","Data":"3131163cb54cc59211946e5dbf54121be8946275ffc5369e130ed4b192a93d80"}
Jul 16 22:40:18.374302 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:18.374188    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" event={"ID":"b352ec86-54b0-41d7-b839-0266d4fe0b87","Type":"ContainerStarted","Data":"8af1dcbdf8d20b1c12dbb93e0aaf4a69c7b5dabe74f21bfdd427589c4f41aecf"}
Jul 16 22:40:18.390604 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:18.390561    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" podStartSLOduration=1.390548041 podStartE2EDuration="1.390548041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:18.390451286 +0000 UTC m=+3174.383615436" watchObservedRunningTime="2026-07-16 22:40:18.390548041 +0000 UTC m=+3174.383712192"
Jul 16 22:40:19.380402 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:19.380368    2555 generic.go:356] "Generic (PLEG): container finished" podID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerID="3131163cb54cc59211946e5dbf54121be8946275ffc5369e130ed4b192a93d80" exitCode=6
Jul 16 22:40:19.380793 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:19.380459    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" event={"ID":"b352ec86-54b0-41d7-b839-0266d4fe0b87","Type":"ContainerDied","Data":"3131163cb54cc59211946e5dbf54121be8946275ffc5369e130ed4b192a93d80"}
Jul 16 22:40:19.380793 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:19.380652    2555 scope.go:117] "RemoveContainer" containerID="3131163cb54cc59211946e5dbf54121be8946275ffc5369e130ed4b192a93d80"
Jul 16 22:40:20.386690 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:20.386653    2555 generic.go:356] "Generic (PLEG): container finished" podID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerID="23d19a4dd22ab0ca84cb23ff68f88bee391d7eef91d669952b91cbc3a5fb0db1" exitCode=6
Jul 16 22:40:20.387135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:20.386741    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" event={"ID":"b352ec86-54b0-41d7-b839-0266d4fe0b87","Type":"ContainerDied","Data":"23d19a4dd22ab0ca84cb23ff68f88bee391d7eef91d669952b91cbc3a5fb0db1"}
Jul 16 22:40:20.387135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:20.386801    2555 scope.go:117] "RemoveContainer" containerID="3131163cb54cc59211946e5dbf54121be8946275ffc5369e130ed4b192a93d80"
Jul 16 22:40:20.387135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:20.386990    2555 scope.go:117] "RemoveContainer" containerID="23d19a4dd22ab0ca84cb23ff68f88bee391d7eef91d669952b91cbc3a5fb0db1"
Jul 16 22:40:20.387290 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:20.387197    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2_odh-ai-gateway-infra(b352ec86-54b0-41d7-b839-0266d4fe0b87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87"
Jul 16 22:40:21.395144 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:21.395116    2555 scope.go:117] "RemoveContainer" containerID="23d19a4dd22ab0ca84cb23ff68f88bee391d7eef91d669952b91cbc3a5fb0db1"
Jul 16 22:40:21.395529 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:21.395278    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2_odh-ai-gateway-infra(b352ec86-54b0-41d7-b839-0266d4fe0b87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87"
Jul 16 22:40:24.491467 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:24.491426    2555 scope.go:117] "RemoveContainer" containerID="775b8ebac07314b870b86d28875a37d77880972d262cb23614f69174ff3db4ec"
Jul 16 22:40:25.417233 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:25.417199    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerID="aa0fa3d338a19097c51aab0879739a9efc0a54b41ce2c846c328b29aa3c82810" exitCode=6
Jul 16 22:40:25.417411 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:25.417278    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" event={"ID":"3c7ac350-405c-4667-961e-f2213f5b96fa","Type":"ContainerDied","Data":"aa0fa3d338a19097c51aab0879739a9efc0a54b41ce2c846c328b29aa3c82810"}
Jul 16 22:40:25.417411 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:25.417354    2555 scope.go:117] "RemoveContainer" containerID="775b8ebac07314b870b86d28875a37d77880972d262cb23614f69174ff3db4ec"
Jul 16 22:40:25.417724 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:25.417707    2555 scope.go:117] "RemoveContainer" containerID="aa0fa3d338a19097c51aab0879739a9efc0a54b41ce2c846c328b29aa3c82810"
Jul 16 22:40:25.417930 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:25.417911    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9_odh-ai-gateway-infra(3c7ac350-405c-4667-961e-f2213f5b96fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa"
Jul 16 22:40:25.489725 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:25.489698    2555 scope.go:117] "RemoveContainer" containerID="3603e7b0dcfed7baff00addcfd077a66bd794909c82e6ee01a6e649e1a8a9592"
Jul 16 22:40:26.423970 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.423921    2555 generic.go:356] "Generic (PLEG): container finished" podID="5d116693-c892-4fbb-bce7-77989aac5855" containerID="731f8dba64acdd169df73bc39237f142a1c4748ec0b42b2a651667914c6ef2ec" exitCode=6
Jul 16 22:40:26.424410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.423994    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" event={"ID":"5d116693-c892-4fbb-bce7-77989aac5855","Type":"ContainerDied","Data":"731f8dba64acdd169df73bc39237f142a1c4748ec0b42b2a651667914c6ef2ec"}
Jul 16 22:40:26.424410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.424063    2555 scope.go:117] "RemoveContainer" containerID="3603e7b0dcfed7baff00addcfd077a66bd794909c82e6ee01a6e649e1a8a9592"
Jul 16 22:40:26.424410 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.424397    2555 scope.go:117] "RemoveContainer" containerID="731f8dba64acdd169df73bc39237f142a1c4748ec0b42b2a651667914c6ef2ec"
Jul 16 22:40:26.424657 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:26.424637    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th_odh-ai-gateway-infra(5d116693-c892-4fbb-bce7-77989aac5855)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" podUID="5d116693-c892-4fbb-bce7-77989aac5855"
Jul 16 22:40:26.449137 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.449103    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"]
Jul 16 22:40:26.577045 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.577018    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"
Jul 16 22:40:26.631211 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.631182    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7ac350-405c-4667-961e-f2213f5b96fa-maas-api-service-ca\") pod \"3c7ac350-405c-4667-961e-f2213f5b96fa\" (UID: \"3c7ac350-405c-4667-961e-f2213f5b96fa\") "
Jul 16 22:40:26.631515 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.631494    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7ac350-405c-4667-961e-f2213f5b96fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c7ac350-405c-4667-961e-f2213f5b96fa" (UID: "3c7ac350-405c-4667-961e-f2213f5b96fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:26.732175 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:26.732139    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7ac350-405c-4667-961e-f2213f5b96fa-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:40:27.431867 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.431837    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"
Jul 16 22:40:27.432385 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.431846    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9" event={"ID":"3c7ac350-405c-4667-961e-f2213f5b96fa","Type":"ContainerDied","Data":"e86c85e155c97072c02d9e7b0ce5f7355f0a1abb83b78c5f93b5a62d5689a955"}
Jul 16 22:40:27.432385 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.431997    2555 scope.go:117] "RemoveContainer" containerID="aa0fa3d338a19097c51aab0879739a9efc0a54b41ce2c846c328b29aa3c82810"
Jul 16 22:40:27.454892 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.454858    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"]
Jul 16 22:40:27.460398 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.460365    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"]
Jul 16 22:40:27.471332 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.471300    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2mdb9"]
Jul 16 22:40:27.595412 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.595387    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"
Jul 16 22:40:27.640309 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.640267    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d116693-c892-4fbb-bce7-77989aac5855-maas-api-service-ca\") pod \"5d116693-c892-4fbb-bce7-77989aac5855\" (UID: \"5d116693-c892-4fbb-bce7-77989aac5855\") "
Jul 16 22:40:27.640636 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.640612    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d116693-c892-4fbb-bce7-77989aac5855-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d116693-c892-4fbb-bce7-77989aac5855" (UID: "5d116693-c892-4fbb-bce7-77989aac5855"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:27.741512 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:27.741422    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d116693-c892-4fbb-bce7-77989aac5855-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:40:28.440582 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:28.440541    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th" event={"ID":"5d116693-c892-4fbb-bce7-77989aac5855","Type":"ContainerDied","Data":"fd68d32d9fa1925be9386cebb1ff09e61e66e8564b6889f5b28dbb1582f3e85b"}
Jul 16 22:40:28.441147 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:28.440608    2555 scope.go:117] "RemoveContainer" containerID="731f8dba64acdd169df73bc39237f142a1c4748ec0b42b2a651667914c6ef2ec"
Jul 16 22:40:28.441147 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:28.440560    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"
Jul 16 22:40:28.461587 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:28.461555    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"]
Jul 16 22:40:28.469572 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:28.469548    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-tn9th"]
Jul 16 22:40:28.494875 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:28.494844    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" path="/var/lib/kubelet/pods/3c7ac350-405c-4667-961e-f2213f5b96fa/volumes"
Jul 16 22:40:28.495226 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:28.495212    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d116693-c892-4fbb-bce7-77989aac5855" path="/var/lib/kubelet/pods/5d116693-c892-4fbb-bce7-77989aac5855/volumes"
Jul 16 22:40:35.489382 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:35.489345    2555 scope.go:117] "RemoveContainer" containerID="23d19a4dd22ab0ca84cb23ff68f88bee391d7eef91d669952b91cbc3a5fb0db1"
Jul 16 22:40:36.478872 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:36.478839    2555 generic.go:356] "Generic (PLEG): container finished" podID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerID="0f44b0e009c3fccbdfdd5d5b07be9c5b65c3b417867a6d432ed76503edf18494" exitCode=6
Jul 16 22:40:36.479074 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:36.478918    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" event={"ID":"b352ec86-54b0-41d7-b839-0266d4fe0b87","Type":"ContainerDied","Data":"0f44b0e009c3fccbdfdd5d5b07be9c5b65c3b417867a6d432ed76503edf18494"}
Jul 16 22:40:36.479074 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:36.479002    2555 scope.go:117] "RemoveContainer" containerID="23d19a4dd22ab0ca84cb23ff68f88bee391d7eef91d669952b91cbc3a5fb0db1"
Jul 16 22:40:36.479316 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:36.479302    2555 scope.go:117] "RemoveContainer" containerID="0f44b0e009c3fccbdfdd5d5b07be9c5b65c3b417867a6d432ed76503edf18494"
Jul 16 22:40:36.479556 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:36.479535    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2_odh-ai-gateway-infra(b352ec86-54b0-41d7-b839-0266d4fe0b87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87"
Jul 16 22:40:37.512813 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:37.512772    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"]
Jul 16 22:40:37.644424 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:37.644401    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"
Jul 16 22:40:37.726594 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:37.726564    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b352ec86-54b0-41d7-b839-0266d4fe0b87-maas-api-service-ca\") pod \"b352ec86-54b0-41d7-b839-0266d4fe0b87\" (UID: \"b352ec86-54b0-41d7-b839-0266d4fe0b87\") "
Jul 16 22:40:37.726928 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:37.726903    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b352ec86-54b0-41d7-b839-0266d4fe0b87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b352ec86-54b0-41d7-b839-0266d4fe0b87" (UID: "b352ec86-54b0-41d7-b839-0266d4fe0b87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:37.827299 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:37.827261    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b352ec86-54b0-41d7-b839-0266d4fe0b87-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:40:38.490081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:38.490048    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"
Jul 16 22:40:38.494417 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:38.494391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2" event={"ID":"b352ec86-54b0-41d7-b839-0266d4fe0b87","Type":"ContainerDied","Data":"8af1dcbdf8d20b1c12dbb93e0aaf4a69c7b5dabe74f21bfdd427589c4f41aecf"}
Jul 16 22:40:38.494583 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:38.494438    2555 scope.go:117] "RemoveContainer" containerID="0f44b0e009c3fccbdfdd5d5b07be9c5b65c3b417867a6d432ed76503edf18494"
Jul 16 22:40:38.515687 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:38.515655    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"]
Jul 16 22:40:38.521699 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:38.521667    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-vtvw2"]
Jul 16 22:40:40.494944 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:40.494898    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" path="/var/lib/kubelet/pods/b352ec86-54b0-41d7-b839-0266d4fe0b87/volumes"
Jul 16 22:40:45.857001 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.856946    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"]
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857249    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857261    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857269    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857275    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857282    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857288    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857297    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857302    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857310    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857315    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857325    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857330    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857336    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857375 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857341    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857388    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857394    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:45.857808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857400    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:45.857808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857406    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:45.857808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857410    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:45.857808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857416    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe051b75-d462-43c0-bf53-cab2380354a1" containerName="revoke-keys"
Jul 16 22:40:45.857808 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.857421    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:45.861799 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.861782    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"
Jul 16 22:40:45.864645 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.864618    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:40:45.864783 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.864651    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:40:45.868457 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.868427    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"]
Jul 16 22:40:45.997075 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:45.997034    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c12d7b2-3068-4091-852a-41a715edf703-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9\" (UID: \"4c12d7b2-3068-4091-852a-41a715edf703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"
Jul 16 22:40:46.098333 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:46.098288    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c12d7b2-3068-4091-852a-41a715edf703-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9\" (UID: \"4c12d7b2-3068-4091-852a-41a715edf703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"
Jul 16 22:40:46.098919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:46.098898    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c12d7b2-3068-4091-852a-41a715edf703-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9\" (UID: \"4c12d7b2-3068-4091-852a-41a715edf703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"
Jul 16 22:40:46.173414 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:46.173314    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"
Jul 16 22:40:46.302143 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:46.302116    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"]
Jul 16 22:40:46.303704 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:40:46.303674    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c12d7b2_3068_4091_852a_41a715edf703.slice/crio-44159765c9e396f4826d48e4c703f9d108fddd0d2c04d3ee437ebf6d484876f8 WatchSource:0}: Error finding container 44159765c9e396f4826d48e4c703f9d108fddd0d2c04d3ee437ebf6d484876f8: Status 404 returned error can't find the container with id 44159765c9e396f4826d48e4c703f9d108fddd0d2c04d3ee437ebf6d484876f8
Jul 16 22:40:46.527834 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:46.527744    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" event={"ID":"4c12d7b2-3068-4091-852a-41a715edf703","Type":"ContainerStarted","Data":"44159765c9e396f4826d48e4c703f9d108fddd0d2c04d3ee437ebf6d484876f8"}
Jul 16 22:40:47.535058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:47.534944    2555 generic.go:356] "Generic (PLEG): container finished" podID="4c12d7b2-3068-4091-852a-41a715edf703" containerID="d49c9796bc26ad0b9eccff9757459e3374bf5262fd73bee4a8bc965bc4e0c045" exitCode=6
Jul 16 22:40:47.535058 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:47.535026    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" event={"ID":"4c12d7b2-3068-4091-852a-41a715edf703","Type":"ContainerDied","Data":"d49c9796bc26ad0b9eccff9757459e3374bf5262fd73bee4a8bc965bc4e0c045"}
Jul 16 22:40:47.535521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:47.535269    2555 scope.go:117] "RemoveContainer" containerID="d49c9796bc26ad0b9eccff9757459e3374bf5262fd73bee4a8bc965bc4e0c045"
Jul 16 22:40:48.541452 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:48.541357    2555 generic.go:356] "Generic (PLEG): container finished" podID="4c12d7b2-3068-4091-852a-41a715edf703" containerID="0f0632ba2bb76b2631219302e431f3d2919b513005abbaa630cc1c88b1b00662" exitCode=6
Jul 16 22:40:48.541452 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:48.541439    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" event={"ID":"4c12d7b2-3068-4091-852a-41a715edf703","Type":"ContainerDied","Data":"0f0632ba2bb76b2631219302e431f3d2919b513005abbaa630cc1c88b1b00662"}
Jul 16 22:40:48.541994 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:48.541503    2555 scope.go:117] "RemoveContainer" containerID="d49c9796bc26ad0b9eccff9757459e3374bf5262fd73bee4a8bc965bc4e0c045"
Jul 16 22:40:48.541994 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:48.541689    2555 scope.go:117] "RemoveContainer" containerID="0f0632ba2bb76b2631219302e431f3d2919b513005abbaa630cc1c88b1b00662"
Jul 16 22:40:48.541994 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:48.541921    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9_odh-ai-gateway-infra(4c12d7b2-3068-4091-852a-41a715edf703)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" podUID="4c12d7b2-3068-4091-852a-41a715edf703"
Jul 16 22:40:49.548108 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:49.548082    2555 scope.go:117] "RemoveContainer" containerID="0f0632ba2bb76b2631219302e431f3d2919b513005abbaa630cc1c88b1b00662"
Jul 16 22:40:49.548517 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:40:49.548255    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9_odh-ai-gateway-infra(4c12d7b2-3068-4091-852a-41a715edf703)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" podUID="4c12d7b2-3068-4091-852a-41a715edf703"
Jul 16 22:40:58.892197 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892164    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"]
Jul 16 22:40:58.892655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892460    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:58.892655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892470    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:58.892655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892483    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:58.892655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892489    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:58.892655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892536    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b352ec86-54b0-41d7-b839-0266d4fe0b87" containerName="revoke-keys"
Jul 16 22:40:58.892655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892546    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d116693-c892-4fbb-bce7-77989aac5855" containerName="revoke-keys"
Jul 16 22:40:58.892655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.892555    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c7ac350-405c-4667-961e-f2213f5b96fa" containerName="revoke-keys"
Jul 16 22:40:58.896637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.896621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"
Jul 16 22:40:58.904460 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:58.904431    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"]
Jul 16 22:40:59.008023 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:59.007950    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caf40af-c8e8-4713-a431-0082bb3153c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz\" (UID: \"8caf40af-c8e8-4713-a431-0082bb3153c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"
Jul 16 22:40:59.109098 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:59.109060    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caf40af-c8e8-4713-a431-0082bb3153c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz\" (UID: \"8caf40af-c8e8-4713-a431-0082bb3153c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"
Jul 16 22:40:59.109687 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:59.109659    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caf40af-c8e8-4713-a431-0082bb3153c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz\" (UID: \"8caf40af-c8e8-4713-a431-0082bb3153c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"
Jul 16 22:40:59.207653 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:59.207561    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"
Jul 16 22:40:59.337850 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:59.337821    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"]
Jul 16 22:40:59.340278 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:40:59.340235    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caf40af_c8e8_4713_a431_0082bb3153c6.slice/crio-fe3d8fe0fec377fdf6bece2b807c58224a4ef9a6dd429d8aa9496059cc709e3d WatchSource:0}: Error finding container fe3d8fe0fec377fdf6bece2b807c58224a4ef9a6dd429d8aa9496059cc709e3d: Status 404 returned error can't find the container with id fe3d8fe0fec377fdf6bece2b807c58224a4ef9a6dd429d8aa9496059cc709e3d
Jul 16 22:40:59.600870 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:40:59.600830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" event={"ID":"8caf40af-c8e8-4713-a431-0082bb3153c6","Type":"ContainerStarted","Data":"fe3d8fe0fec377fdf6bece2b807c58224a4ef9a6dd429d8aa9496059cc709e3d"}
Jul 16 22:41:00.607032 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:00.606997    2555 generic.go:356] "Generic (PLEG): container finished" podID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerID="baadc1068c7234031c19d956c5c4eace690a1c5dea38833f9aac734f3c0238d0" exitCode=6
Jul 16 22:41:00.607411 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:00.607080    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" event={"ID":"8caf40af-c8e8-4713-a431-0082bb3153c6","Type":"ContainerDied","Data":"baadc1068c7234031c19d956c5c4eace690a1c5dea38833f9aac734f3c0238d0"}
Jul 16 22:41:00.607411 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:00.607233    2555 scope.go:117] "RemoveContainer" containerID="baadc1068c7234031c19d956c5c4eace690a1c5dea38833f9aac734f3c0238d0"
Jul 16 22:41:01.490411 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:01.490336    2555 scope.go:117] "RemoveContainer" containerID="0f0632ba2bb76b2631219302e431f3d2919b513005abbaa630cc1c88b1b00662"
Jul 16 22:41:01.613713 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:01.613673    2555 generic.go:356] "Generic (PLEG): container finished" podID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerID="b7bce25b5120b1302f9299b86ef5b92111e04a8990619b90e386e789e82c9d59" exitCode=6
Jul 16 22:41:01.614208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:01.613749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" event={"ID":"8caf40af-c8e8-4713-a431-0082bb3153c6","Type":"ContainerDied","Data":"b7bce25b5120b1302f9299b86ef5b92111e04a8990619b90e386e789e82c9d59"}
Jul 16 22:41:01.614208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:01.613822    2555 scope.go:117] "RemoveContainer" containerID="baadc1068c7234031c19d956c5c4eace690a1c5dea38833f9aac734f3c0238d0"
Jul 16 22:41:01.614208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:01.614018    2555 scope.go:117] "RemoveContainer" containerID="b7bce25b5120b1302f9299b86ef5b92111e04a8990619b90e386e789e82c9d59"
Jul 16 22:41:01.614321 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:01.614243    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz_odh-ai-gateway-infra(8caf40af-c8e8-4713-a431-0082bb3153c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6"
Jul 16 22:41:02.619892 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:02.619862    2555 generic.go:356] "Generic (PLEG): container finished" podID="4c12d7b2-3068-4091-852a-41a715edf703" containerID="aec18ee02a0376396628aabdc60d9aad5df438c60436c179dc1e974fc2ccb6ad" exitCode=6
Jul 16 22:41:02.620406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:02.619944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" event={"ID":"4c12d7b2-3068-4091-852a-41a715edf703","Type":"ContainerDied","Data":"aec18ee02a0376396628aabdc60d9aad5df438c60436c179dc1e974fc2ccb6ad"}
Jul 16 22:41:02.620406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:02.620044    2555 scope.go:117] "RemoveContainer" containerID="0f0632ba2bb76b2631219302e431f3d2919b513005abbaa630cc1c88b1b00662"
Jul 16 22:41:02.620406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:02.620293    2555 scope.go:117] "RemoveContainer" containerID="aec18ee02a0376396628aabdc60d9aad5df438c60436c179dc1e974fc2ccb6ad"
Jul 16 22:41:02.620608 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:02.620538    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9_odh-ai-gateway-infra(4c12d7b2-3068-4091-852a-41a715edf703)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" podUID="4c12d7b2-3068-4091-852a-41a715edf703"
Jul 16 22:41:02.621672 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:02.621653    2555 scope.go:117] "RemoveContainer" containerID="b7bce25b5120b1302f9299b86ef5b92111e04a8990619b90e386e789e82c9d59"
Jul 16 22:41:02.621837 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:02.621820    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz_odh-ai-gateway-infra(8caf40af-c8e8-4713-a431-0082bb3153c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6"
Jul 16 22:41:03.651069 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:03.651038    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"]
Jul 16 22:41:03.784452 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:03.784427    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"
Jul 16 22:41:03.849255 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:03.849224    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c12d7b2-3068-4091-852a-41a715edf703-maas-api-service-ca\") pod \"4c12d7b2-3068-4091-852a-41a715edf703\" (UID: \"4c12d7b2-3068-4091-852a-41a715edf703\") "
Jul 16 22:41:03.849590 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:03.849570    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c12d7b2-3068-4091-852a-41a715edf703-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c12d7b2-3068-4091-852a-41a715edf703" (UID: "4c12d7b2-3068-4091-852a-41a715edf703"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:03.950475 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:03.950394    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c12d7b2-3068-4091-852a-41a715edf703-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:41:04.632174 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:04.632146    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"
Jul 16 22:41:04.632174 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:04.632169    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9" event={"ID":"4c12d7b2-3068-4091-852a-41a715edf703","Type":"ContainerDied","Data":"44159765c9e396f4826d48e4c703f9d108fddd0d2c04d3ee437ebf6d484876f8"}
Jul 16 22:41:04.632427 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:04.632225    2555 scope.go:117] "RemoveContainer" containerID="aec18ee02a0376396628aabdc60d9aad5df438c60436c179dc1e974fc2ccb6ad"
Jul 16 22:41:04.649468 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:04.649440    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"]
Jul 16 22:41:04.653444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:04.653422    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-gxrh9"]
Jul 16 22:41:06.494689 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.494644    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c12d7b2-3068-4091-852a-41a715edf703" path="/var/lib/kubelet/pods/4c12d7b2-3068-4091-852a-41a715edf703/volumes"
Jul 16 22:41:06.508841 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.508811    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"]
Jul 16 22:41:06.509114 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509102    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.509158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509116    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.509158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509134    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.509158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509139    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.509158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509146    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.509158 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509152    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.509321 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509203    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.509321 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.509210    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:06.512037 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.512022    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"
Jul 16 22:41:06.520815 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.520790    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"]
Jul 16 22:41:06.675555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.675519    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6839970d-d4aa-4fd5-8150-5c78de3f9a72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72\" (UID: \"6839970d-d4aa-4fd5-8150-5c78de3f9a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"
Jul 16 22:41:06.776277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.776190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6839970d-d4aa-4fd5-8150-5c78de3f9a72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72\" (UID: \"6839970d-d4aa-4fd5-8150-5c78de3f9a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"
Jul 16 22:41:06.776807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.776787    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6839970d-d4aa-4fd5-8150-5c78de3f9a72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72\" (UID: \"6839970d-d4aa-4fd5-8150-5c78de3f9a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"
Jul 16 22:41:06.822454 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.822419    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"
Jul 16 22:41:06.954677 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:06.954652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"]
Jul 16 22:41:06.956504 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:41:06.956477    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6839970d_d4aa_4fd5_8150_5c78de3f9a72.slice/crio-bd64996ede9d6266b7857b4b1094ad59f5214dc80fa26ab5284942cbc524428c WatchSource:0}: Error finding container bd64996ede9d6266b7857b4b1094ad59f5214dc80fa26ab5284942cbc524428c: Status 404 returned error can't find the container with id bd64996ede9d6266b7857b4b1094ad59f5214dc80fa26ab5284942cbc524428c
Jul 16 22:41:07.651816 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:07.651784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" event={"ID":"6839970d-d4aa-4fd5-8150-5c78de3f9a72","Type":"ContainerStarted","Data":"e20da7186cc8a8b528b8e5b39c8076700b24eaa4504d5eeae52925f6170fcb54"}
Jul 16 22:41:07.652225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:07.651845    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" event={"ID":"6839970d-d4aa-4fd5-8150-5c78de3f9a72","Type":"ContainerStarted","Data":"bd64996ede9d6266b7857b4b1094ad59f5214dc80fa26ab5284942cbc524428c"}
Jul 16 22:41:07.670589 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:07.670538    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" podStartSLOduration=1.670520781 podStartE2EDuration="1.670520781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:07.669536922 +0000 UTC m=+3223.662701074" watchObservedRunningTime="2026-07-16 22:41:07.670520781 +0000 UTC m=+3223.663684933"
Jul 16 22:41:08.658231 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:08.658193    2555 generic.go:356] "Generic (PLEG): container finished" podID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerID="e20da7186cc8a8b528b8e5b39c8076700b24eaa4504d5eeae52925f6170fcb54" exitCode=6
Jul 16 22:41:08.658673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:08.658241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" event={"ID":"6839970d-d4aa-4fd5-8150-5c78de3f9a72","Type":"ContainerDied","Data":"e20da7186cc8a8b528b8e5b39c8076700b24eaa4504d5eeae52925f6170fcb54"}
Jul 16 22:41:08.658673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:08.658482    2555 scope.go:117] "RemoveContainer" containerID="e20da7186cc8a8b528b8e5b39c8076700b24eaa4504d5eeae52925f6170fcb54"
Jul 16 22:41:09.665051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:09.665015    2555 generic.go:356] "Generic (PLEG): container finished" podID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerID="d54a99b6e3e9f4b1d5381c6563bc3f35120bb0026db135e29e55cd635c2ebb2c" exitCode=6
Jul 16 22:41:09.665472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:09.665094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" event={"ID":"6839970d-d4aa-4fd5-8150-5c78de3f9a72","Type":"ContainerDied","Data":"d54a99b6e3e9f4b1d5381c6563bc3f35120bb0026db135e29e55cd635c2ebb2c"}
Jul 16 22:41:09.665472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:09.665157    2555 scope.go:117] "RemoveContainer" containerID="e20da7186cc8a8b528b8e5b39c8076700b24eaa4504d5eeae52925f6170fcb54"
Jul 16 22:41:09.665472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:09.665388    2555 scope.go:117] "RemoveContainer" containerID="d54a99b6e3e9f4b1d5381c6563bc3f35120bb0026db135e29e55cd635c2ebb2c"
Jul 16 22:41:09.665656 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:09.665601    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72_odh-ai-gateway-infra(6839970d-d4aa-4fd5-8150-5c78de3f9a72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72"
Jul 16 22:41:10.671242 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:10.671219    2555 scope.go:117] "RemoveContainer" containerID="d54a99b6e3e9f4b1d5381c6563bc3f35120bb0026db135e29e55cd635c2ebb2c"
Jul 16 22:41:10.671598 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:10.671415    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72_odh-ai-gateway-infra(6839970d-d4aa-4fd5-8150-5c78de3f9a72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72"
Jul 16 22:41:15.489605 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:15.489573    2555 scope.go:117] "RemoveContainer" containerID="b7bce25b5120b1302f9299b86ef5b92111e04a8990619b90e386e789e82c9d59"
Jul 16 22:41:15.695493 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:15.695457    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" event={"ID":"8caf40af-c8e8-4713-a431-0082bb3153c6","Type":"ContainerStarted","Data":"a5cab3c3f4cf1a4c02d4b5d70c1fe3e3ebc1bf91d1ad58287e1c501df0d4b369"}
Jul 16 22:41:15.711795 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:15.711745    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" podStartSLOduration=17.711729328 podStartE2EDuration="17.711729328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:15.710366152 +0000 UTC m=+3231.703530317" watchObservedRunningTime="2026-07-16 22:41:15.711729328 +0000 UTC m=+3231.704893480"
Jul 16 22:41:16.701283 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:16.701243    2555 generic.go:356] "Generic (PLEG): container finished" podID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerID="a5cab3c3f4cf1a4c02d4b5d70c1fe3e3ebc1bf91d1ad58287e1c501df0d4b369" exitCode=6
Jul 16 22:41:16.701714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:16.701311    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" event={"ID":"8caf40af-c8e8-4713-a431-0082bb3153c6","Type":"ContainerDied","Data":"a5cab3c3f4cf1a4c02d4b5d70c1fe3e3ebc1bf91d1ad58287e1c501df0d4b369"}
Jul 16 22:41:16.701714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:16.701395    2555 scope.go:117] "RemoveContainer" containerID="b7bce25b5120b1302f9299b86ef5b92111e04a8990619b90e386e789e82c9d59"
Jul 16 22:41:16.701714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:16.701628    2555 scope.go:117] "RemoveContainer" containerID="a5cab3c3f4cf1a4c02d4b5d70c1fe3e3ebc1bf91d1ad58287e1c501df0d4b369"
Jul 16 22:41:16.701889 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:16.701858    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz_odh-ai-gateway-infra(8caf40af-c8e8-4713-a431-0082bb3153c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6"
Jul 16 22:41:16.722605 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:16.722576    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"]
Jul 16 22:41:17.637408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.637367    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"]
Jul 16 22:41:17.637758 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.637745    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c12d7b2-3068-4091-852a-41a715edf703" containerName="revoke-keys"
Jul 16 22:41:17.642051 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.642025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"
Jul 16 22:41:17.655213 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.655187    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"]
Jul 16 22:41:17.767907 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.767867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1380e8a9-4575-4eb3-946c-f191ed8f68aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2\" (UID: \"1380e8a9-4575-4eb3-946c-f191ed8f68aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"
Jul 16 22:41:17.839044 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.839017    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"
Jul 16 22:41:17.868949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.868921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1380e8a9-4575-4eb3-946c-f191ed8f68aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2\" (UID: \"1380e8a9-4575-4eb3-946c-f191ed8f68aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"
Jul 16 22:41:17.869525 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.869501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1380e8a9-4575-4eb3-946c-f191ed8f68aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2\" (UID: \"1380e8a9-4575-4eb3-946c-f191ed8f68aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"
Jul 16 22:41:17.951253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.951164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"
Jul 16 22:41:17.970225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.970198    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caf40af-c8e8-4713-a431-0082bb3153c6-maas-api-service-ca\") pod \"8caf40af-c8e8-4713-a431-0082bb3153c6\" (UID: \"8caf40af-c8e8-4713-a431-0082bb3153c6\") "
Jul 16 22:41:17.970556 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:17.970526    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8caf40af-c8e8-4713-a431-0082bb3153c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8caf40af-c8e8-4713-a431-0082bb3153c6" (UID: "8caf40af-c8e8-4713-a431-0082bb3153c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:18.071022 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.070995    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caf40af-c8e8-4713-a431-0082bb3153c6-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:41:18.081163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.081135    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"]
Jul 16 22:41:18.082900 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:41:18.082878    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1380e8a9_4575_4eb3_946c_f191ed8f68aa.slice/crio-b5c15e739d16825ca43bc59c551bbe5d025f56cc24261591299173bbb9ceb4bb WatchSource:0}: Error finding container b5c15e739d16825ca43bc59c551bbe5d025f56cc24261591299173bbb9ceb4bb: Status 404 returned error can't find the container with id b5c15e739d16825ca43bc59c551bbe5d025f56cc24261591299173bbb9ceb4bb
Jul 16 22:41:18.713987 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.713941    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"
Jul 16 22:41:18.713987 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.713972    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz" event={"ID":"8caf40af-c8e8-4713-a431-0082bb3153c6","Type":"ContainerDied","Data":"fe3d8fe0fec377fdf6bece2b807c58224a4ef9a6dd429d8aa9496059cc709e3d"}
Jul 16 22:41:18.714220 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.714035    2555 scope.go:117] "RemoveContainer" containerID="a5cab3c3f4cf1a4c02d4b5d70c1fe3e3ebc1bf91d1ad58287e1c501df0d4b369"
Jul 16 22:41:18.715603 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.715580    2555 generic.go:356] "Generic (PLEG): container finished" podID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerID="2e295c5346d26432ac2df7bf4402d049479a41a00b219f7880dc4874ad853809" exitCode=6
Jul 16 22:41:18.715704 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.715662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" event={"ID":"1380e8a9-4575-4eb3-946c-f191ed8f68aa","Type":"ContainerDied","Data":"2e295c5346d26432ac2df7bf4402d049479a41a00b219f7880dc4874ad853809"}
Jul 16 22:41:18.715761 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.715709    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" event={"ID":"1380e8a9-4575-4eb3-946c-f191ed8f68aa","Type":"ContainerStarted","Data":"b5c15e739d16825ca43bc59c551bbe5d025f56cc24261591299173bbb9ceb4bb"}
Jul 16 22:41:18.715863 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.715843    2555 scope.go:117] "RemoveContainer" containerID="2e295c5346d26432ac2df7bf4402d049479a41a00b219f7880dc4874ad853809"
Jul 16 22:41:18.759356 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.759315    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"]
Jul 16 22:41:18.760901 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:18.760874    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-mpvzz"]
Jul 16 22:41:19.723152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:19.723068    2555 generic.go:356] "Generic (PLEG): container finished" podID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerID="98b5b289ceecb6860ad4402cba622b12cf3758c6283a49ced276c70cde734958" exitCode=6
Jul 16 22:41:19.723152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:19.723107    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" event={"ID":"1380e8a9-4575-4eb3-946c-f191ed8f68aa","Type":"ContainerDied","Data":"98b5b289ceecb6860ad4402cba622b12cf3758c6283a49ced276c70cde734958"}
Jul 16 22:41:19.723630 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:19.723159    2555 scope.go:117] "RemoveContainer" containerID="2e295c5346d26432ac2df7bf4402d049479a41a00b219f7880dc4874ad853809"
Jul 16 22:41:19.723630 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:19.723329    2555 scope.go:117] "RemoveContainer" containerID="98b5b289ceecb6860ad4402cba622b12cf3758c6283a49ced276c70cde734958"
Jul 16 22:41:19.723630 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:19.723561    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2_odh-ai-gateway-infra(1380e8a9-4575-4eb3-946c-f191ed8f68aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa"
Jul 16 22:41:20.494882 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:20.494851    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" path="/var/lib/kubelet/pods/8caf40af-c8e8-4713-a431-0082bb3153c6/volumes"
Jul 16 22:41:20.729485 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:20.729460    2555 scope.go:117] "RemoveContainer" containerID="98b5b289ceecb6860ad4402cba622b12cf3758c6283a49ced276c70cde734958"
Jul 16 22:41:20.729876 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:20.729637    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2_odh-ai-gateway-infra(1380e8a9-4575-4eb3-946c-f191ed8f68aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa"
Jul 16 22:41:23.489718 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:23.489681    2555 scope.go:117] "RemoveContainer" containerID="d54a99b6e3e9f4b1d5381c6563bc3f35120bb0026db135e29e55cd635c2ebb2c"
Jul 16 22:41:24.752309 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:24.752276    2555 generic.go:356] "Generic (PLEG): container finished" podID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerID="25fdc53ce1ccec1e3e683b0b45ee3060746b2f21f6ec069e779e85e4b873774c" exitCode=6
Jul 16 22:41:24.752729 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:24.752346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" event={"ID":"6839970d-d4aa-4fd5-8150-5c78de3f9a72","Type":"ContainerDied","Data":"25fdc53ce1ccec1e3e683b0b45ee3060746b2f21f6ec069e779e85e4b873774c"}
Jul 16 22:41:24.752729 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:24.752404    2555 scope.go:117] "RemoveContainer" containerID="d54a99b6e3e9f4b1d5381c6563bc3f35120bb0026db135e29e55cd635c2ebb2c"
Jul 16 22:41:24.752729 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:24.752671    2555 scope.go:117] "RemoveContainer" containerID="25fdc53ce1ccec1e3e683b0b45ee3060746b2f21f6ec069e779e85e4b873774c"
Jul 16 22:41:24.752925 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:24.752887    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72_odh-ai-gateway-infra(6839970d-d4aa-4fd5-8150-5c78de3f9a72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72"
Jul 16 22:41:25.793077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:25.793028    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"]
Jul 16 22:41:25.924464 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:25.924440    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"
Jul 16 22:41:25.931323 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:25.931304    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6839970d-d4aa-4fd5-8150-5c78de3f9a72-maas-api-service-ca\") pod \"6839970d-d4aa-4fd5-8150-5c78de3f9a72\" (UID: \"6839970d-d4aa-4fd5-8150-5c78de3f9a72\") "
Jul 16 22:41:25.931622 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:25.931602    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6839970d-d4aa-4fd5-8150-5c78de3f9a72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6839970d-d4aa-4fd5-8150-5c78de3f9a72" (UID: "6839970d-d4aa-4fd5-8150-5c78de3f9a72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:26.032398 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:26.032367    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6839970d-d4aa-4fd5-8150-5c78de3f9a72-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:41:26.763616 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:26.763533    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"
Jul 16 22:41:26.763616 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:26.763544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72" event={"ID":"6839970d-d4aa-4fd5-8150-5c78de3f9a72","Type":"ContainerDied","Data":"bd64996ede9d6266b7857b4b1094ad59f5214dc80fa26ab5284942cbc524428c"}
Jul 16 22:41:26.763616 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:26.763599    2555 scope.go:117] "RemoveContainer" containerID="25fdc53ce1ccec1e3e683b0b45ee3060746b2f21f6ec069e779e85e4b873774c"
Jul 16 22:41:26.785336 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:26.785306    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"]
Jul 16 22:41:26.789750 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:26.789726    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xxv72"]
Jul 16 22:41:28.494592 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:28.494553    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" path="/var/lib/kubelet/pods/6839970d-d4aa-4fd5-8150-5c78de3f9a72/volumes"
Jul 16 22:41:35.490298 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.490265    2555 scope.go:117] "RemoveContainer" containerID="98b5b289ceecb6860ad4402cba622b12cf3758c6283a49ced276c70cde734958"
Jul 16 22:41:35.809011 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.808907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" event={"ID":"1380e8a9-4575-4eb3-946c-f191ed8f68aa","Type":"ContainerStarted","Data":"7d0138fa3caa6c45e7031bbb54e3484b4935ea61eee7450402dff0f0aea0b57e"}
Jul 16 22:41:35.833557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.833503    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" podStartSLOduration=18.833487359 podStartE2EDuration="18.833487359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:35.832200811 +0000 UTC m=+3251.825364976" watchObservedRunningTime="2026-07-16 22:41:35.833487359 +0000 UTC m=+3251.826651510"
Jul 16 22:41:35.908636 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908598    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"]
Jul 16 22:41:35.908905 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908894    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:35.908971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908907    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:35.908971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908922    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:35.908971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908928    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:35.908971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908935    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:35.908971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908940    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:35.908971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908946    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:35.908971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.908951    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:35.909202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.909023    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:35.909202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.909030    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:35.909202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.909036    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:35.909202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.909041    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:35.909202 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.909047    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:35.911835 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.911820    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"
Jul 16 22:41:35.921794 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:35.921758    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"]
Jul 16 22:41:36.015506 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.015464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k\" (UID: \"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"
Jul 16 22:41:36.116717 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.116684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k\" (UID: \"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"
Jul 16 22:41:36.117320 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.117302    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k\" (UID: \"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"
Jul 16 22:41:36.251682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.251644    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"
Jul 16 22:41:36.396860 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.396831    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"]
Jul 16 22:41:36.397929 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:41:36.397903    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7d7f1d_9ddf_4ecf_8c1b_f8d6cf0b187b.slice/crio-6ebc6b3dcffc7a74ad23b426855875b3da7cc550a62c6ae0736684ab2c773584 WatchSource:0}: Error finding container 6ebc6b3dcffc7a74ad23b426855875b3da7cc550a62c6ae0736684ab2c773584: Status 404 returned error can't find the container with id 6ebc6b3dcffc7a74ad23b426855875b3da7cc550a62c6ae0736684ab2c773584
Jul 16 22:41:36.815135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.815052    2555 generic.go:356] "Generic (PLEG): container finished" podID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerID="7d0138fa3caa6c45e7031bbb54e3484b4935ea61eee7450402dff0f0aea0b57e" exitCode=6
Jul 16 22:41:36.815135 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.815123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" event={"ID":"1380e8a9-4575-4eb3-946c-f191ed8f68aa","Type":"ContainerDied","Data":"7d0138fa3caa6c45e7031bbb54e3484b4935ea61eee7450402dff0f0aea0b57e"}
Jul 16 22:41:36.815655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.815185    2555 scope.go:117] "RemoveContainer" containerID="98b5b289ceecb6860ad4402cba622b12cf3758c6283a49ced276c70cde734958"
Jul 16 22:41:36.815655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.815510    2555 scope.go:117] "RemoveContainer" containerID="7d0138fa3caa6c45e7031bbb54e3484b4935ea61eee7450402dff0f0aea0b57e"
Jul 16 22:41:36.815759 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:36.815729    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2_odh-ai-gateway-infra(1380e8a9-4575-4eb3-946c-f191ed8f68aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa"
Jul 16 22:41:36.816684 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.816659    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" event={"ID":"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b","Type":"ContainerStarted","Data":"afb2fb29b919d57b0687a6c6bdff90041909399f258b2dec56b4486129e1e045"}
Jul 16 22:41:36.816801 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.816699    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" event={"ID":"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b","Type":"ContainerStarted","Data":"6ebc6b3dcffc7a74ad23b426855875b3da7cc550a62c6ae0736684ab2c773584"}
Jul 16 22:41:36.845194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.845164    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"]
Jul 16 22:41:36.856227 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:36.856179    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" podStartSLOduration=1.856163701 podStartE2EDuration="1.856163701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:36.854722315 +0000 UTC m=+3252.847886468" watchObservedRunningTime="2026-07-16 22:41:36.856163701 +0000 UTC m=+3252.849327852"
Jul 16 22:41:37.824227 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:37.824193    2555 generic.go:356] "Generic (PLEG): container finished" podID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerID="afb2fb29b919d57b0687a6c6bdff90041909399f258b2dec56b4486129e1e045" exitCode=6
Jul 16 22:41:37.824619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:37.824270    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" event={"ID":"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b","Type":"ContainerDied","Data":"afb2fb29b919d57b0687a6c6bdff90041909399f258b2dec56b4486129e1e045"}
Jul 16 22:41:37.824619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:37.824462    2555 scope.go:117] "RemoveContainer" containerID="afb2fb29b919d57b0687a6c6bdff90041909399f258b2dec56b4486129e1e045"
Jul 16 22:41:37.954920 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:37.954898    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"
Jul 16 22:41:38.030819 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.030791    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1380e8a9-4575-4eb3-946c-f191ed8f68aa-maas-api-service-ca\") pod \"1380e8a9-4575-4eb3-946c-f191ed8f68aa\" (UID: \"1380e8a9-4575-4eb3-946c-f191ed8f68aa\") "
Jul 16 22:41:38.031189 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.031161    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1380e8a9-4575-4eb3-946c-f191ed8f68aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1380e8a9-4575-4eb3-946c-f191ed8f68aa" (UID: "1380e8a9-4575-4eb3-946c-f191ed8f68aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:38.132033 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.131997    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1380e8a9-4575-4eb3-946c-f191ed8f68aa-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:41:38.830846 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.830820    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"
Jul 16 22:41:38.831340 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.830818    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2" event={"ID":"1380e8a9-4575-4eb3-946c-f191ed8f68aa","Type":"ContainerDied","Data":"b5c15e739d16825ca43bc59c551bbe5d025f56cc24261591299173bbb9ceb4bb"}
Jul 16 22:41:38.831340 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.830997    2555 scope.go:117] "RemoveContainer" containerID="7d0138fa3caa6c45e7031bbb54e3484b4935ea61eee7450402dff0f0aea0b57e"
Jul 16 22:41:38.832785 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.832756    2555 generic.go:356] "Generic (PLEG): container finished" podID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerID="03d8aa9a257ddabc88ba8681904fe30cce42f86955961c8b3d3c15e99d95f30b" exitCode=6
Jul 16 22:41:38.832921 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.832897    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" event={"ID":"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b","Type":"ContainerDied","Data":"03d8aa9a257ddabc88ba8681904fe30cce42f86955961c8b3d3c15e99d95f30b"}
Jul 16 22:41:38.833153 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.833134    2555 scope.go:117] "RemoveContainer" containerID="03d8aa9a257ddabc88ba8681904fe30cce42f86955961c8b3d3c15e99d95f30b"
Jul 16 22:41:38.833391 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:38.833364    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k_odh-ai-gateway-infra(0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b"
Jul 16 22:41:38.840542 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.840524    2555 scope.go:117] "RemoveContainer" containerID="afb2fb29b919d57b0687a6c6bdff90041909399f258b2dec56b4486129e1e045"
Jul 16 22:41:38.872462 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.872432    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"]
Jul 16 22:41:38.879749 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:38.879722    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-nz9w2"]
Jul 16 22:41:39.845021 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:39.844993    2555 scope.go:117] "RemoveContainer" containerID="03d8aa9a257ddabc88ba8681904fe30cce42f86955961c8b3d3c15e99d95f30b"
Jul 16 22:41:39.845409 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:39.845237    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k_odh-ai-gateway-infra(0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b"
Jul 16 22:41:40.494989 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:40.494927    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" path="/var/lib/kubelet/pods/1380e8a9-4575-4eb3-946c-f191ed8f68aa/volumes"
Jul 16 22:41:48.966075 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966039    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"]
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966345    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966356    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caf40af-c8e8-4713-a431-0082bb3153c6" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966364    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966370    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966377    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966383    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966390    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966395    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966402    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966407    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966462    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966469    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6839970d-d4aa-4fd5-8150-5c78de3f9a72" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966476    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.966547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.966482    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1380e8a9-4575-4eb3-946c-f191ed8f68aa" containerName="revoke-keys"
Jul 16 22:41:48.970796 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.970777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"
Jul 16 22:41:48.978137 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:48.978113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"]
Jul 16 22:41:49.034951 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.034913    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e601078a-ed25-4c2b-af8e-6507c6f2da21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-ltv92\" (UID: \"e601078a-ed25-4c2b-af8e-6507c6f2da21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"
Jul 16 22:41:49.135930 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.135896    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e601078a-ed25-4c2b-af8e-6507c6f2da21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-ltv92\" (UID: \"e601078a-ed25-4c2b-af8e-6507c6f2da21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"
Jul 16 22:41:49.136566 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.136547    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e601078a-ed25-4c2b-af8e-6507c6f2da21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-ltv92\" (UID: \"e601078a-ed25-4c2b-af8e-6507c6f2da21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"
Jul 16 22:41:49.281614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.281529    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"
Jul 16 22:41:49.426615 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.426582    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"]
Jul 16 22:41:49.430042 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:41:49.430017    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode601078a_ed25_4c2b_af8e_6507c6f2da21.slice/crio-de205bc324ab5047a48a0be0572e22447f490ac5e23dba4396f5459549f66029 WatchSource:0}: Error finding container de205bc324ab5047a48a0be0572e22447f490ac5e23dba4396f5459549f66029: Status 404 returned error can't find the container with id de205bc324ab5047a48a0be0572e22447f490ac5e23dba4396f5459549f66029
Jul 16 22:41:49.892041 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.892001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" event={"ID":"e601078a-ed25-4c2b-af8e-6507c6f2da21","Type":"ContainerStarted","Data":"eddc06c3b3efe360d2b863a2e2106c8150d3839308f2b3849cd3e50881035084"}
Jul 16 22:41:49.892221 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.892052    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" event={"ID":"e601078a-ed25-4c2b-af8e-6507c6f2da21","Type":"ContainerStarted","Data":"de205bc324ab5047a48a0be0572e22447f490ac5e23dba4396f5459549f66029"}
Jul 16 22:41:49.927134 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:49.927027    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" podStartSLOduration=1.9270127110000002 podStartE2EDuration="1.927012711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:49.926707643 +0000 UTC m=+3265.919871796" watchObservedRunningTime="2026-07-16 22:41:49.927012711 +0000 UTC m=+3265.920176916"
Jul 16 22:41:50.904313 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:50.904274    2555 generic.go:356] "Generic (PLEG): container finished" podID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerID="eddc06c3b3efe360d2b863a2e2106c8150d3839308f2b3849cd3e50881035084" exitCode=6
Jul 16 22:41:50.904754 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:50.904357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" event={"ID":"e601078a-ed25-4c2b-af8e-6507c6f2da21","Type":"ContainerDied","Data":"eddc06c3b3efe360d2b863a2e2106c8150d3839308f2b3849cd3e50881035084"}
Jul 16 22:41:50.904754 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:50.904532    2555 scope.go:117] "RemoveContainer" containerID="eddc06c3b3efe360d2b863a2e2106c8150d3839308f2b3849cd3e50881035084"
Jul 16 22:41:51.910648 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:51.910613    2555 generic.go:356] "Generic (PLEG): container finished" podID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerID="bb48fb116dedf2b3eaaa34337b5a90adfe7c1abf513ebdc9c0577808ddf6deb9" exitCode=6
Jul 16 22:41:51.911145 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:51.910702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" event={"ID":"e601078a-ed25-4c2b-af8e-6507c6f2da21","Type":"ContainerDied","Data":"bb48fb116dedf2b3eaaa34337b5a90adfe7c1abf513ebdc9c0577808ddf6deb9"}
Jul 16 22:41:51.911145 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:51.910764    2555 scope.go:117] "RemoveContainer" containerID="eddc06c3b3efe360d2b863a2e2106c8150d3839308f2b3849cd3e50881035084"
Jul 16 22:41:51.911145 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:51.910972    2555 scope.go:117] "RemoveContainer" containerID="bb48fb116dedf2b3eaaa34337b5a90adfe7c1abf513ebdc9c0577808ddf6deb9"
Jul 16 22:41:51.911262 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:51.911187    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-ltv92_odh-ai-gateway-infra(e601078a-ed25-4c2b-af8e-6507c6f2da21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21"
Jul 16 22:41:52.916139 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:52.916107    2555 scope.go:117] "RemoveContainer" containerID="bb48fb116dedf2b3eaaa34337b5a90adfe7c1abf513ebdc9c0577808ddf6deb9"
Jul 16 22:41:52.916606 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:52.916350    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-ltv92_odh-ai-gateway-infra(e601078a-ed25-4c2b-af8e-6507c6f2da21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21"
Jul 16 22:41:55.490414 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:55.490384    2555 scope.go:117] "RemoveContainer" containerID="03d8aa9a257ddabc88ba8681904fe30cce42f86955961c8b3d3c15e99d95f30b"
Jul 16 22:41:55.931594 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:55.931562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" event={"ID":"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b","Type":"ContainerStarted","Data":"ae506984dc58614cd02d0966425b0479fd6ea7931392f10f4f4b08d1f6e7c8a8"}
Jul 16 22:41:56.938186 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:56.938149    2555 generic.go:356] "Generic (PLEG): container finished" podID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerID="ae506984dc58614cd02d0966425b0479fd6ea7931392f10f4f4b08d1f6e7c8a8" exitCode=6
Jul 16 22:41:56.938682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:56.938220    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" event={"ID":"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b","Type":"ContainerDied","Data":"ae506984dc58614cd02d0966425b0479fd6ea7931392f10f4f4b08d1f6e7c8a8"}
Jul 16 22:41:56.938682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:56.938296    2555 scope.go:117] "RemoveContainer" containerID="03d8aa9a257ddabc88ba8681904fe30cce42f86955961c8b3d3c15e99d95f30b"
Jul 16 22:41:56.938682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:56.938638    2555 scope.go:117] "RemoveContainer" containerID="ae506984dc58614cd02d0966425b0479fd6ea7931392f10f4f4b08d1f6e7c8a8"
Jul 16 22:41:56.938882 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:41:56.938860    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k_odh-ai-gateway-infra(0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b"
Jul 16 22:41:56.972193 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:56.972149    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"]
Jul 16 22:41:58.080612 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.080590    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"
Jul 16 22:41:58.218469 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.218372    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b-maas-api-service-ca\") pod \"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b\" (UID: \"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b\") "
Jul 16 22:41:58.218778 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.218749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" (UID: "0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:58.319557 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.319523    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:41:58.951131 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.951098    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k" event={"ID":"0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b","Type":"ContainerDied","Data":"6ebc6b3dcffc7a74ad23b426855875b3da7cc550a62c6ae0736684ab2c773584"}
Jul 16 22:41:58.951131 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.951131    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"
Jul 16 22:41:58.951379 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.951166    2555 scope.go:117] "RemoveContainer" containerID="ae506984dc58614cd02d0966425b0479fd6ea7931392f10f4f4b08d1f6e7c8a8"
Jul 16 22:41:58.969002 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.968972    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"]
Jul 16 22:41:58.977193 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:41:58.977166    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-x9f5k"]
Jul 16 22:42:00.494866 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:00.494834    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" path="/var/lib/kubelet/pods/0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b/volumes"
Jul 16 22:42:03.489973 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:03.489935    2555 scope.go:117] "RemoveContainer" containerID="bb48fb116dedf2b3eaaa34337b5a90adfe7c1abf513ebdc9c0577808ddf6deb9"
Jul 16 22:42:03.975795 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:03.975758    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" event={"ID":"e601078a-ed25-4c2b-af8e-6507c6f2da21","Type":"ContainerStarted","Data":"9c8d1af856e72d8f8c008a40becdff6625e4268236a27c89d92b6590cf77e471"}
Jul 16 22:42:04.981654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:04.981619    2555 generic.go:356] "Generic (PLEG): container finished" podID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerID="9c8d1af856e72d8f8c008a40becdff6625e4268236a27c89d92b6590cf77e471" exitCode=6
Jul 16 22:42:04.982091 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:04.981691    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" event={"ID":"e601078a-ed25-4c2b-af8e-6507c6f2da21","Type":"ContainerDied","Data":"9c8d1af856e72d8f8c008a40becdff6625e4268236a27c89d92b6590cf77e471"}
Jul 16 22:42:04.982091 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:04.981755    2555 scope.go:117] "RemoveContainer" containerID="bb48fb116dedf2b3eaaa34337b5a90adfe7c1abf513ebdc9c0577808ddf6deb9"
Jul 16 22:42:04.982091 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:04.982019    2555 scope.go:117] "RemoveContainer" containerID="9c8d1af856e72d8f8c008a40becdff6625e4268236a27c89d92b6590cf77e471"
Jul 16 22:42:04.982286 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:04.982265    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-ltv92_odh-ai-gateway-infra(e601078a-ed25-4c2b-af8e-6507c6f2da21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21"
Jul 16 22:42:05.053813 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:05.053781    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"]
Jul 16 22:42:06.113256 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.113232    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"
Jul 16 22:42:06.288320 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.288225    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e601078a-ed25-4c2b-af8e-6507c6f2da21-maas-api-service-ca\") pod \"e601078a-ed25-4c2b-af8e-6507c6f2da21\" (UID: \"e601078a-ed25-4c2b-af8e-6507c6f2da21\") "
Jul 16 22:42:06.288626 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.288605    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e601078a-ed25-4c2b-af8e-6507c6f2da21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e601078a-ed25-4c2b-af8e-6507c6f2da21" (UID: "e601078a-ed25-4c2b-af8e-6507c6f2da21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:06.389028 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.388991    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e601078a-ed25-4c2b-af8e-6507c6f2da21-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:42:06.623935 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.623902    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"]
Jul 16 22:42:06.624239 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624225    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:06.624289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624241    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:06.624289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624252    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:06.624289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624260    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:06.624289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624271    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:06.624289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624278    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:06.624289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624287    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:06.624289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624292    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:06.624499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624351    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:06.624499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624359    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:06.624499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624364    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:06.624499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624372    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:06.624499 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.624377    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:06.627325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.627308    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"
Jul 16 22:42:06.649356 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.649325    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"]
Jul 16 22:42:06.792476 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.792441    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c7b73c-d18c-46b1-8fe8-e0204b318aca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6\" (UID: \"d3c7b73c-d18c-46b1-8fe8-e0204b318aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"
Jul 16 22:42:06.893206 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.893133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c7b73c-d18c-46b1-8fe8-e0204b318aca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6\" (UID: \"d3c7b73c-d18c-46b1-8fe8-e0204b318aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"
Jul 16 22:42:06.893737 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.893718    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c7b73c-d18c-46b1-8fe8-e0204b318aca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6\" (UID: \"d3c7b73c-d18c-46b1-8fe8-e0204b318aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"
Jul 16 22:42:06.937290 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.937258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"
Jul 16 22:42:06.994559 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.994509    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"
Jul 16 22:42:06.994703 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.994562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92" event={"ID":"e601078a-ed25-4c2b-af8e-6507c6f2da21","Type":"ContainerDied","Data":"de205bc324ab5047a48a0be0572e22447f490ac5e23dba4396f5459549f66029"}
Jul 16 22:42:06.994703 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:06.994628    2555 scope.go:117] "RemoveContainer" containerID="9c8d1af856e72d8f8c008a40becdff6625e4268236a27c89d92b6590cf77e471"
Jul 16 22:42:07.014846 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:07.014819    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"]
Jul 16 22:42:07.018172 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:07.018148    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ltv92"]
Jul 16 22:42:07.079748 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:07.079723    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"]
Jul 16 22:42:07.081803 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:42:07.081777    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c7b73c_d18c_46b1_8fe8_e0204b318aca.slice/crio-48046a608ce9ce716800a5285068f9dae4223e8393c34c62dbd1c46616639e78 WatchSource:0}: Error finding container 48046a608ce9ce716800a5285068f9dae4223e8393c34c62dbd1c46616639e78: Status 404 returned error can't find the container with id 48046a608ce9ce716800a5285068f9dae4223e8393c34c62dbd1c46616639e78
Jul 16 22:42:08.001637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:08.001543    2555 generic.go:356] "Generic (PLEG): container finished" podID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerID="54f38cd601b9ff7b58302b78860e8f1a139d6fcddc6f7ccfc4ce3df9c80fce06" exitCode=6
Jul 16 22:42:08.001637 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:08.001608    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" event={"ID":"d3c7b73c-d18c-46b1-8fe8-e0204b318aca","Type":"ContainerDied","Data":"54f38cd601b9ff7b58302b78860e8f1a139d6fcddc6f7ccfc4ce3df9c80fce06"}
Jul 16 22:42:08.002132 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:08.001649    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" event={"ID":"d3c7b73c-d18c-46b1-8fe8-e0204b318aca","Type":"ContainerStarted","Data":"48046a608ce9ce716800a5285068f9dae4223e8393c34c62dbd1c46616639e78"}
Jul 16 22:42:08.002132 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:08.001850    2555 scope.go:117] "RemoveContainer" containerID="54f38cd601b9ff7b58302b78860e8f1a139d6fcddc6f7ccfc4ce3df9c80fce06"
Jul 16 22:42:08.494813 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:08.494775    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" path="/var/lib/kubelet/pods/e601078a-ed25-4c2b-af8e-6507c6f2da21/volumes"
Jul 16 22:42:09.008558 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:09.008466    2555 generic.go:356] "Generic (PLEG): container finished" podID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerID="40c5b4b3ddb1f81acaa95bae34a12cde9d6e0195aa87265ac1875021efcd309f" exitCode=6
Jul 16 22:42:09.008558 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:09.008512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" event={"ID":"d3c7b73c-d18c-46b1-8fe8-e0204b318aca","Type":"ContainerDied","Data":"40c5b4b3ddb1f81acaa95bae34a12cde9d6e0195aa87265ac1875021efcd309f"}
Jul 16 22:42:09.008558 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:09.008561    2555 scope.go:117] "RemoveContainer" containerID="54f38cd601b9ff7b58302b78860e8f1a139d6fcddc6f7ccfc4ce3df9c80fce06"
Jul 16 22:42:09.009143 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:09.008791    2555 scope.go:117] "RemoveContainer" containerID="40c5b4b3ddb1f81acaa95bae34a12cde9d6e0195aa87265ac1875021efcd309f"
Jul 16 22:42:09.009143 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:09.009047    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6_odh-ai-gateway-infra(d3c7b73c-d18c-46b1-8fe8-e0204b318aca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca"
Jul 16 22:42:10.014371 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:10.014338    2555 scope.go:117] "RemoveContainer" containerID="40c5b4b3ddb1f81acaa95bae34a12cde9d6e0195aa87265ac1875021efcd309f"
Jul 16 22:42:10.014838 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:10.014582    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6_odh-ai-gateway-infra(d3c7b73c-d18c-46b1-8fe8-e0204b318aca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca"
Jul 16 22:42:17.714817 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.714777    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"]
Jul 16 22:42:17.715230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.715128    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:17.715230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.715141    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7d7f1d-9ddf-4ecf-8c1b-f8d6cf0b187b" containerName="revoke-keys"
Jul 16 22:42:17.715230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.715153    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:17.715230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.715158    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:17.715230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.715222    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e601078a-ed25-4c2b-af8e-6507c6f2da21" containerName="revoke-keys"
Jul 16 22:42:17.719366 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.719350    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"
Jul 16 22:42:17.737267 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.737238    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"]
Jul 16 22:42:17.780804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.780766    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b96d5-a2a7-49ce-90f6-f6f42e0640ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7\" (UID: \"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"
Jul 16 22:42:17.882074 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.882031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b96d5-a2a7-49ce-90f6-f6f42e0640ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7\" (UID: \"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"
Jul 16 22:42:17.882643 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:17.882623    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b96d5-a2a7-49ce-90f6-f6f42e0640ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7\" (UID: \"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"
Jul 16 22:42:18.028948 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:18.028870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"
Jul 16 22:42:18.170025 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:18.169994    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"]
Jul 16 22:42:18.170650 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:42:18.170627    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67b96d5_a2a7_49ce_90f6_f6f42e0640ec.slice/crio-8e1e72ff487baa60e65c9ac3be7b319a3d62073b55707de9109507156c3cd478 WatchSource:0}: Error finding container 8e1e72ff487baa60e65c9ac3be7b319a3d62073b55707de9109507156c3cd478: Status 404 returned error can't find the container with id 8e1e72ff487baa60e65c9ac3be7b319a3d62073b55707de9109507156c3cd478
Jul 16 22:42:19.056974 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:19.056864    2555 generic.go:356] "Generic (PLEG): container finished" podID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerID="bfcedcb335737e932b70bada4c2ef6d547aad50138d44497752f00ecb75a9bf7" exitCode=6
Jul 16 22:42:19.056974 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:19.056908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" event={"ID":"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec","Type":"ContainerDied","Data":"bfcedcb335737e932b70bada4c2ef6d547aad50138d44497752f00ecb75a9bf7"}
Jul 16 22:42:19.057377 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:19.056980    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" event={"ID":"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec","Type":"ContainerStarted","Data":"8e1e72ff487baa60e65c9ac3be7b319a3d62073b55707de9109507156c3cd478"}
Jul 16 22:42:19.057377 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:19.057160    2555 scope.go:117] "RemoveContainer" containerID="bfcedcb335737e932b70bada4c2ef6d547aad50138d44497752f00ecb75a9bf7"
Jul 16 22:42:20.062900 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:20.062811    2555 generic.go:356] "Generic (PLEG): container finished" podID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerID="7b3bc032b146555648154ee8236d61eb30b75f57ef7ba9d1429f682b3dbcb609" exitCode=6
Jul 16 22:42:20.063439 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:20.062894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" event={"ID":"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec","Type":"ContainerDied","Data":"7b3bc032b146555648154ee8236d61eb30b75f57ef7ba9d1429f682b3dbcb609"}
Jul 16 22:42:20.063439 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:20.062984    2555 scope.go:117] "RemoveContainer" containerID="bfcedcb335737e932b70bada4c2ef6d547aad50138d44497752f00ecb75a9bf7"
Jul 16 22:42:20.063439 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:20.063143    2555 scope.go:117] "RemoveContainer" containerID="7b3bc032b146555648154ee8236d61eb30b75f57ef7ba9d1429f682b3dbcb609"
Jul 16 22:42:20.063439 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:20.063349    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7_odh-ai-gateway-infra(c67b96d5-a2a7-49ce-90f6-f6f42e0640ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec"
Jul 16 22:42:21.069232 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:21.069206    2555 scope.go:117] "RemoveContainer" containerID="7b3bc032b146555648154ee8236d61eb30b75f57ef7ba9d1429f682b3dbcb609"
Jul 16 22:42:21.069715 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:21.069470    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7_odh-ai-gateway-infra(c67b96d5-a2a7-49ce-90f6-f6f42e0640ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec"
Jul 16 22:42:21.490045 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:21.490017    2555 scope.go:117] "RemoveContainer" containerID="40c5b4b3ddb1f81acaa95bae34a12cde9d6e0195aa87265ac1875021efcd309f"
Jul 16 22:42:22.075353 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:22.075271    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" event={"ID":"d3c7b73c-d18c-46b1-8fe8-e0204b318aca","Type":"ContainerStarted","Data":"08bb3905c16a6bff9506648cf3ac31b5d11f3596b3239bb03a32925cc2db861d"}
Jul 16 22:42:22.093241 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:22.093194    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" podStartSLOduration=16.093178776 podStartE2EDuration="16.093178776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:22.091925089 +0000 UTC m=+3298.085089241" watchObservedRunningTime="2026-07-16 22:42:22.093178776 +0000 UTC m=+3298.086342978"
Jul 16 22:42:23.082479 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:23.082442    2555 generic.go:356] "Generic (PLEG): container finished" podID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerID="08bb3905c16a6bff9506648cf3ac31b5d11f3596b3239bb03a32925cc2db861d" exitCode=6
Jul 16 22:42:23.082479 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:23.082487    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" event={"ID":"d3c7b73c-d18c-46b1-8fe8-e0204b318aca","Type":"ContainerDied","Data":"08bb3905c16a6bff9506648cf3ac31b5d11f3596b3239bb03a32925cc2db861d"}
Jul 16 22:42:23.082983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:23.082533    2555 scope.go:117] "RemoveContainer" containerID="40c5b4b3ddb1f81acaa95bae34a12cde9d6e0195aa87265ac1875021efcd309f"
Jul 16 22:42:23.082983 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:23.082812    2555 scope.go:117] "RemoveContainer" containerID="08bb3905c16a6bff9506648cf3ac31b5d11f3596b3239bb03a32925cc2db861d"
Jul 16 22:42:23.083078 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:23.083015    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6_odh-ai-gateway-infra(d3c7b73c-d18c-46b1-8fe8-e0204b318aca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca"
Jul 16 22:42:23.114428 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:23.114397    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"]
Jul 16 22:42:24.227311 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:24.227290    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"
Jul 16 22:42:24.230127 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:24.230110    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c7b73c-d18c-46b1-8fe8-e0204b318aca-maas-api-service-ca\") pod \"d3c7b73c-d18c-46b1-8fe8-e0204b318aca\" (UID: \"d3c7b73c-d18c-46b1-8fe8-e0204b318aca\") "
Jul 16 22:42:24.230414 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:24.230394    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c7b73c-d18c-46b1-8fe8-e0204b318aca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3c7b73c-d18c-46b1-8fe8-e0204b318aca" (UID: "d3c7b73c-d18c-46b1-8fe8-e0204b318aca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:24.330908 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:24.330872    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c7b73c-d18c-46b1-8fe8-e0204b318aca-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:42:25.100614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:25.100571    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"
Jul 16 22:42:25.100614 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:25.100573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6" event={"ID":"d3c7b73c-d18c-46b1-8fe8-e0204b318aca","Type":"ContainerDied","Data":"48046a608ce9ce716800a5285068f9dae4223e8393c34c62dbd1c46616639e78"}
Jul 16 22:42:25.100827 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:25.100721    2555 scope.go:117] "RemoveContainer" containerID="08bb3905c16a6bff9506648cf3ac31b5d11f3596b3239bb03a32925cc2db861d"
Jul 16 22:42:25.123577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:25.123537    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"]
Jul 16 22:42:25.128271 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:25.128239    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-hg7n6"]
Jul 16 22:42:26.494736 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:26.494700    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" path="/var/lib/kubelet/pods/d3c7b73c-d18c-46b1-8fe8-e0204b318aca/volumes"
Jul 16 22:42:35.490258 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.490218    2555 scope.go:117] "RemoveContainer" containerID="7b3bc032b146555648154ee8236d61eb30b75f57ef7ba9d1429f682b3dbcb609"
Jul 16 22:42:35.969247 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969211    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"]
Jul 16 22:42:35.969519 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969507    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969521    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969529    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969535    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969550    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969565 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969555    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969605    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969613    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.969733 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.969621    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c7b73c-d18c-46b1-8fe8-e0204b318aca" containerName="revoke-keys"
Jul 16 22:42:35.972563 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.972547    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"
Jul 16 22:42:35.984495 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:35.984467    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"]
Jul 16 22:42:36.019255 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.019212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5792ccab-63c0-4a0a-9a4c-5629926636b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw\" (UID: \"5792ccab-63c0-4a0a-9a4c-5629926636b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"
Jul 16 22:42:36.120543 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.120507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5792ccab-63c0-4a0a-9a4c-5629926636b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw\" (UID: \"5792ccab-63c0-4a0a-9a4c-5629926636b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"
Jul 16 22:42:36.121177 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.121156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5792ccab-63c0-4a0a-9a4c-5629926636b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw\" (UID: \"5792ccab-63c0-4a0a-9a4c-5629926636b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"
Jul 16 22:42:36.155664 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.155632    2555 generic.go:356] "Generic (PLEG): container finished" podID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerID="3f0f12910e43ab8390a45e62cc3dd62ab757734ea519e425243331c87af2c1c7" exitCode=6
Jul 16 22:42:36.155788 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.155675    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" event={"ID":"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec","Type":"ContainerDied","Data":"3f0f12910e43ab8390a45e62cc3dd62ab757734ea519e425243331c87af2c1c7"}
Jul 16 22:42:36.155788 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.155728    2555 scope.go:117] "RemoveContainer" containerID="7b3bc032b146555648154ee8236d61eb30b75f57ef7ba9d1429f682b3dbcb609"
Jul 16 22:42:36.156070 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.156046    2555 scope.go:117] "RemoveContainer" containerID="3f0f12910e43ab8390a45e62cc3dd62ab757734ea519e425243331c87af2c1c7"
Jul 16 22:42:36.156320 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:36.156296    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7_odh-ai-gateway-infra(c67b96d5-a2a7-49ce-90f6-f6f42e0640ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec"
Jul 16 22:42:36.283128 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.283044    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"
Jul 16 22:42:36.624464 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:36.624437    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"]
Jul 16 22:42:36.625510 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:42:36.625477    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5792ccab_63c0_4a0a_9a4c_5629926636b7.slice/crio-b3495b256b7e57a36494e8286c3416a5cb8c55052e06c905b63a56cc8d65fa73 WatchSource:0}: Error finding container b3495b256b7e57a36494e8286c3416a5cb8c55052e06c905b63a56cc8d65fa73: Status 404 returned error can't find the container with id b3495b256b7e57a36494e8286c3416a5cb8c55052e06c905b63a56cc8d65fa73
Jul 16 22:42:37.162465 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.162428    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" event={"ID":"5792ccab-63c0-4a0a-9a4c-5629926636b7","Type":"ContainerStarted","Data":"4576b670a78ce45ee911557c5cd5457244163a7f7dd4837f68af58ee0d9782f8"}
Jul 16 22:42:37.162665 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.162477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" event={"ID":"5792ccab-63c0-4a0a-9a4c-5629926636b7","Type":"ContainerStarted","Data":"b3495b256b7e57a36494e8286c3416a5cb8c55052e06c905b63a56cc8d65fa73"}
Jul 16 22:42:37.193310 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.193257    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" podStartSLOduration=2.19324049 podStartE2EDuration="2.19324049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:37.192402953 +0000 UTC m=+3313.185567107" watchObservedRunningTime="2026-07-16 22:42:37.19324049 +0000 UTC m=+3313.186404641"
Jul 16 22:42:37.198194 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.198130    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"]
Jul 16 22:42:37.331893 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.331869    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"
Jul 16 22:42:37.432508 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.432478    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b96d5-a2a7-49ce-90f6-f6f42e0640ec-maas-api-service-ca\") pod \"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec\" (UID: \"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec\") "
Jul 16 22:42:37.432805 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.432782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67b96d5-a2a7-49ce-90f6-f6f42e0640ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" (UID: "c67b96d5-a2a7-49ce-90f6-f6f42e0640ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:37.533378 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:37.533346    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b96d5-a2a7-49ce-90f6-f6f42e0640ec-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:42:38.168588 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.168560    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"
Jul 16 22:42:38.169073 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.168604    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7" event={"ID":"c67b96d5-a2a7-49ce-90f6-f6f42e0640ec","Type":"ContainerDied","Data":"8e1e72ff487baa60e65c9ac3be7b319a3d62073b55707de9109507156c3cd478"}
Jul 16 22:42:38.169073 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.168689    2555 scope.go:117] "RemoveContainer" containerID="3f0f12910e43ab8390a45e62cc3dd62ab757734ea519e425243331c87af2c1c7"
Jul 16 22:42:38.170280 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.170254    2555 generic.go:356] "Generic (PLEG): container finished" podID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerID="4576b670a78ce45ee911557c5cd5457244163a7f7dd4837f68af58ee0d9782f8" exitCode=6
Jul 16 22:42:38.170370 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.170312    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" event={"ID":"5792ccab-63c0-4a0a-9a4c-5629926636b7","Type":"ContainerDied","Data":"4576b670a78ce45ee911557c5cd5457244163a7f7dd4837f68af58ee0d9782f8"}
Jul 16 22:42:38.170521 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.170509    2555 scope.go:117] "RemoveContainer" containerID="4576b670a78ce45ee911557c5cd5457244163a7f7dd4837f68af58ee0d9782f8"
Jul 16 22:42:38.221092 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.221059    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"]
Jul 16 22:42:38.226811 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.226781    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xs8k7"]
Jul 16 22:42:38.494883 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:38.494789    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" path="/var/lib/kubelet/pods/c67b96d5-a2a7-49ce-90f6-f6f42e0640ec/volumes"
Jul 16 22:42:39.027355 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.027279    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"]
Jul 16 22:42:39.027577 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.027566    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:39.027623 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.027579    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:39.027623 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.027603    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:39.027623 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.027608    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:39.027714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.027656    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:39.027714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.027664    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:39.030312 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.030295    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"
Jul 16 22:42:39.051188 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.051165    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"]
Jul 16 22:42:39.148770 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.148734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3170d5-1a46-4411-9908-c4d424ad9a1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-56h8n\" (UID: \"eb3170d5-1a46-4411-9908-c4d424ad9a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"
Jul 16 22:42:39.176895 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.176866    2555 generic.go:356] "Generic (PLEG): container finished" podID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerID="0ca681e5fa5b67db31e81122d82ca58ab3057160fed6c134fdb63420ab114054" exitCode=6
Jul 16 22:42:39.177335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.176979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" event={"ID":"5792ccab-63c0-4a0a-9a4c-5629926636b7","Type":"ContainerDied","Data":"0ca681e5fa5b67db31e81122d82ca58ab3057160fed6c134fdb63420ab114054"}
Jul 16 22:42:39.177335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.177042    2555 scope.go:117] "RemoveContainer" containerID="4576b670a78ce45ee911557c5cd5457244163a7f7dd4837f68af58ee0d9782f8"
Jul 16 22:42:39.177335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.177165    2555 scope.go:117] "RemoveContainer" containerID="0ca681e5fa5b67db31e81122d82ca58ab3057160fed6c134fdb63420ab114054"
Jul 16 22:42:39.177486 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:39.177394    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw_odh-ai-gateway-infra(5792ccab-63c0-4a0a-9a4c-5629926636b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7"
Jul 16 22:42:39.249545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.249512    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3170d5-1a46-4411-9908-c4d424ad9a1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-56h8n\" (UID: \"eb3170d5-1a46-4411-9908-c4d424ad9a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"
Jul 16 22:42:39.250151 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.250134    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3170d5-1a46-4411-9908-c4d424ad9a1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-56h8n\" (UID: \"eb3170d5-1a46-4411-9908-c4d424ad9a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"
Jul 16 22:42:39.339855 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.339816    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"
Jul 16 22:42:39.675689 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:39.675665    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"]
Jul 16 22:42:39.678051 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:42:39.678022    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3170d5_1a46_4411_9908_c4d424ad9a1b.slice/crio-dcc618607ae1ee83d410212a224665d8f983074d606334ab3cd72fca5a4e5f69 WatchSource:0}: Error finding container dcc618607ae1ee83d410212a224665d8f983074d606334ab3cd72fca5a4e5f69: Status 404 returned error can't find the container with id dcc618607ae1ee83d410212a224665d8f983074d606334ab3cd72fca5a4e5f69
Jul 16 22:42:40.183991 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:40.183932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" event={"ID":"eb3170d5-1a46-4411-9908-c4d424ad9a1b","Type":"ContainerStarted","Data":"89ba6e4f2eb50244b3e5a2ec9c6fd250b6657f4683170389b4a4938e2e20f1ad"}
Jul 16 22:42:40.184458 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:40.184015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" event={"ID":"eb3170d5-1a46-4411-9908-c4d424ad9a1b","Type":"ContainerStarted","Data":"dcc618607ae1ee83d410212a224665d8f983074d606334ab3cd72fca5a4e5f69"}
Jul 16 22:42:40.185661 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:40.185640    2555 scope.go:117] "RemoveContainer" containerID="0ca681e5fa5b67db31e81122d82ca58ab3057160fed6c134fdb63420ab114054"
Jul 16 22:42:40.185867 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:40.185847    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw_odh-ai-gateway-infra(5792ccab-63c0-4a0a-9a4c-5629926636b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7"
Jul 16 22:42:40.203551 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:40.203472    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" podStartSLOduration=2.203461127 podStartE2EDuration="2.203461127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:40.202021348 +0000 UTC m=+3316.195185495" watchObservedRunningTime="2026-07-16 22:42:40.203461127 +0000 UTC m=+3316.196625304"
Jul 16 22:42:41.191421 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:41.191387    2555 generic.go:356] "Generic (PLEG): container finished" podID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerID="89ba6e4f2eb50244b3e5a2ec9c6fd250b6657f4683170389b4a4938e2e20f1ad" exitCode=6
Jul 16 22:42:41.191807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:41.191446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" event={"ID":"eb3170d5-1a46-4411-9908-c4d424ad9a1b","Type":"ContainerDied","Data":"89ba6e4f2eb50244b3e5a2ec9c6fd250b6657f4683170389b4a4938e2e20f1ad"}
Jul 16 22:42:41.191807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:41.191637    2555 scope.go:117] "RemoveContainer" containerID="89ba6e4f2eb50244b3e5a2ec9c6fd250b6657f4683170389b4a4938e2e20f1ad"
Jul 16 22:42:42.197469 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:42.197434    2555 generic.go:356] "Generic (PLEG): container finished" podID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerID="cf6c6271ee1115c90f3693d35380bcf0014354b14fd4c68a0e50e3858cc4246a" exitCode=6
Jul 16 22:42:42.197927 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:42.197528    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" event={"ID":"eb3170d5-1a46-4411-9908-c4d424ad9a1b","Type":"ContainerDied","Data":"cf6c6271ee1115c90f3693d35380bcf0014354b14fd4c68a0e50e3858cc4246a"}
Jul 16 22:42:42.197927 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:42.197606    2555 scope.go:117] "RemoveContainer" containerID="89ba6e4f2eb50244b3e5a2ec9c6fd250b6657f4683170389b4a4938e2e20f1ad"
Jul 16 22:42:42.197927 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:42.197718    2555 scope.go:117] "RemoveContainer" containerID="cf6c6271ee1115c90f3693d35380bcf0014354b14fd4c68a0e50e3858cc4246a"
Jul 16 22:42:42.198099 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:42.197936    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-56h8n_odh-ai-gateway-infra(eb3170d5-1a46-4411-9908-c4d424ad9a1b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b"
Jul 16 22:42:43.203077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:43.203043    2555 scope.go:117] "RemoveContainer" containerID="cf6c6271ee1115c90f3693d35380bcf0014354b14fd4c68a0e50e3858cc4246a"
Jul 16 22:42:43.203434 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:43.203231    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-56h8n_odh-ai-gateway-infra(eb3170d5-1a46-4411-9908-c4d424ad9a1b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b"
Jul 16 22:42:54.492216 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:54.492186    2555 scope.go:117] "RemoveContainer" containerID="0ca681e5fa5b67db31e81122d82ca58ab3057160fed6c134fdb63420ab114054"
Jul 16 22:42:55.262705 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:55.262673    2555 generic.go:356] "Generic (PLEG): container finished" podID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerID="807a419478ec5d2760867a8185cc68ddac379161aede0ceb7b995205357f4410" exitCode=6
Jul 16 22:42:55.262917 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:55.262750    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" event={"ID":"5792ccab-63c0-4a0a-9a4c-5629926636b7","Type":"ContainerDied","Data":"807a419478ec5d2760867a8185cc68ddac379161aede0ceb7b995205357f4410"}
Jul 16 22:42:55.262917 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:55.262822    2555 scope.go:117] "RemoveContainer" containerID="0ca681e5fa5b67db31e81122d82ca58ab3057160fed6c134fdb63420ab114054"
Jul 16 22:42:55.263160 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:55.263139    2555 scope.go:117] "RemoveContainer" containerID="807a419478ec5d2760867a8185cc68ddac379161aede0ceb7b995205357f4410"
Jul 16 22:42:55.263393 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:55.263373    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw_odh-ai-gateway-infra(5792ccab-63c0-4a0a-9a4c-5629926636b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7"
Jul 16 22:42:56.290485 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.290450    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"]
Jul 16 22:42:56.439421 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.439393    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"
Jul 16 22:42:56.515085 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.515055    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5792ccab-63c0-4a0a-9a4c-5629926636b7-maas-api-service-ca\") pod \"5792ccab-63c0-4a0a-9a4c-5629926636b7\" (UID: \"5792ccab-63c0-4a0a-9a4c-5629926636b7\") "
Jul 16 22:42:56.515416 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.515394    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5792ccab-63c0-4a0a-9a4c-5629926636b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5792ccab-63c0-4a0a-9a4c-5629926636b7" (UID: "5792ccab-63c0-4a0a-9a4c-5629926636b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:56.616030 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.615994    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5792ccab-63c0-4a0a-9a4c-5629926636b7-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:42:56.662635 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.662598    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"]
Jul 16 22:42:56.662936 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.662922    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:56.663008 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.662938    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:56.663008 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.662973    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663008 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.662980    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663008 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.662990    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663008 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.663000    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663008 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.663007    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663201 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.663012    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663201 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.663061    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663201 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.663071    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663201 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.663083    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" containerName="revoke-keys"
Jul 16 22:42:56.663201 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.663090    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67b96d5-a2a7-49ce-90f6-f6f42e0640ec" containerName="revoke-keys"
Jul 16 22:42:56.666152 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.666128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"
Jul 16 22:42:56.676714 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.676688    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"]
Jul 16 22:42:56.817533 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.817493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05001318-156f-41de-adc3-6e4dcf9e8ca6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx\" (UID: \"05001318-156f-41de-adc3-6e4dcf9e8ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"
Jul 16 22:42:56.918676 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.918576    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05001318-156f-41de-adc3-6e4dcf9e8ca6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx\" (UID: \"05001318-156f-41de-adc3-6e4dcf9e8ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"
Jul 16 22:42:56.919292 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.919267    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05001318-156f-41de-adc3-6e4dcf9e8ca6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx\" (UID: \"05001318-156f-41de-adc3-6e4dcf9e8ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"
Jul 16 22:42:56.976802 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:56.976772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"
Jul 16 22:42:57.108704 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.108678    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"]
Jul 16 22:42:57.110264 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:42:57.110228    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05001318_156f_41de_adc3_6e4dcf9e8ca6.slice/crio-95272055f3573a6c8c057d22da9c33108bf96259eb691d9d1aa8ab08f5495e9c WatchSource:0}: Error finding container 95272055f3573a6c8c057d22da9c33108bf96259eb691d9d1aa8ab08f5495e9c: Status 404 returned error can't find the container with id 95272055f3573a6c8c057d22da9c33108bf96259eb691d9d1aa8ab08f5495e9c
Jul 16 22:42:57.274747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.274717    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" event={"ID":"05001318-156f-41de-adc3-6e4dcf9e8ca6","Type":"ContainerStarted","Data":"df20d963b45fe60e18074896736ad94ef65ab24e09b6c7902f00794459841e22"}
Jul 16 22:42:57.274866 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.274773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" event={"ID":"05001318-156f-41de-adc3-6e4dcf9e8ca6","Type":"ContainerStarted","Data":"95272055f3573a6c8c057d22da9c33108bf96259eb691d9d1aa8ab08f5495e9c"}
Jul 16 22:42:57.276010 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.275989    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw" event={"ID":"5792ccab-63c0-4a0a-9a4c-5629926636b7","Type":"ContainerDied","Data":"b3495b256b7e57a36494e8286c3416a5cb8c55052e06c905b63a56cc8d65fa73"}
Jul 16 22:42:57.276103 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.276037    2555 scope.go:117] "RemoveContainer" containerID="807a419478ec5d2760867a8185cc68ddac379161aede0ceb7b995205357f4410"
Jul 16 22:42:57.276103 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.275996    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"
Jul 16 22:42:57.293495 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.293456    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" podStartSLOduration=1.2934429760000001 podStartE2EDuration="1.293442976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:57.291510523 +0000 UTC m=+3333.284674689" watchObservedRunningTime="2026-07-16 22:42:57.293442976 +0000 UTC m=+3333.286607128"
Jul 16 22:42:57.311253 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.311229    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"]
Jul 16 22:42:57.317747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:57.317726    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-xhlcw"]
Jul 16 22:42:58.281475 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:58.281433    2555 generic.go:356] "Generic (PLEG): container finished" podID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerID="df20d963b45fe60e18074896736ad94ef65ab24e09b6c7902f00794459841e22" exitCode=6
Jul 16 22:42:58.281682 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:58.281519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" event={"ID":"05001318-156f-41de-adc3-6e4dcf9e8ca6","Type":"ContainerDied","Data":"df20d963b45fe60e18074896736ad94ef65ab24e09b6c7902f00794459841e22"}
Jul 16 22:42:58.281765 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:58.281717    2555 scope.go:117] "RemoveContainer" containerID="df20d963b45fe60e18074896736ad94ef65ab24e09b6c7902f00794459841e22"
Jul 16 22:42:58.490294 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:58.490260    2555 scope.go:117] "RemoveContainer" containerID="cf6c6271ee1115c90f3693d35380bcf0014354b14fd4c68a0e50e3858cc4246a"
Jul 16 22:42:58.495235 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:58.495207    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5792ccab-63c0-4a0a-9a4c-5629926636b7" path="/var/lib/kubelet/pods/5792ccab-63c0-4a0a-9a4c-5629926636b7/volumes"
Jul 16 22:42:59.288855 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.288762    2555 generic.go:356] "Generic (PLEG): container finished" podID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerID="5d127f8b7f2a63a91b99bb2d357b6c98bf43c782d41ffbab9d53f768302028d6" exitCode=6
Jul 16 22:42:59.288855 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.288837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" event={"ID":"eb3170d5-1a46-4411-9908-c4d424ad9a1b","Type":"ContainerDied","Data":"5d127f8b7f2a63a91b99bb2d357b6c98bf43c782d41ffbab9d53f768302028d6"}
Jul 16 22:42:59.289096 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.288889    2555 scope.go:117] "RemoveContainer" containerID="cf6c6271ee1115c90f3693d35380bcf0014354b14fd4c68a0e50e3858cc4246a"
Jul 16 22:42:59.289271 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.289245    2555 scope.go:117] "RemoveContainer" containerID="5d127f8b7f2a63a91b99bb2d357b6c98bf43c782d41ffbab9d53f768302028d6"
Jul 16 22:42:59.289488 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:59.289463    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-56h8n_odh-ai-gateway-infra(eb3170d5-1a46-4411-9908-c4d424ad9a1b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b"
Jul 16 22:42:59.290747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.290727    2555 generic.go:356] "Generic (PLEG): container finished" podID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerID="5e7d54ea7a3011ab2d9094b1a726c96ddce997b4b08366c0ed7f031134127184" exitCode=6
Jul 16 22:42:59.290824 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.290799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" event={"ID":"05001318-156f-41de-adc3-6e4dcf9e8ca6","Type":"ContainerDied","Data":"5e7d54ea7a3011ab2d9094b1a726c96ddce997b4b08366c0ed7f031134127184"}
Jul 16 22:42:59.290975 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.290939    2555 scope.go:117] "RemoveContainer" containerID="5e7d54ea7a3011ab2d9094b1a726c96ddce997b4b08366c0ed7f031134127184"
Jul 16 22:42:59.291159 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:42:59.291142    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx_odh-ai-gateway-infra(05001318-156f-41de-adc3-6e4dcf9e8ca6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6"
Jul 16 22:42:59.299613 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:42:59.299594    2555 scope.go:117] "RemoveContainer" containerID="df20d963b45fe60e18074896736ad94ef65ab24e09b6c7902f00794459841e22"
Jul 16 22:43:00.297787 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:00.297758    2555 scope.go:117] "RemoveContainer" containerID="5e7d54ea7a3011ab2d9094b1a726c96ddce997b4b08366c0ed7f031134127184"
Jul 16 22:43:00.298197 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:00.297950    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx_odh-ai-gateway-infra(05001318-156f-41de-adc3-6e4dcf9e8ca6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6"
Jul 16 22:43:00.335636 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:00.335610    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"]
Jul 16 22:43:00.468505 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:00.468483    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"
Jul 16 22:43:00.552369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:00.552261    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3170d5-1a46-4411-9908-c4d424ad9a1b-maas-api-service-ca\") pod \"eb3170d5-1a46-4411-9908-c4d424ad9a1b\" (UID: \"eb3170d5-1a46-4411-9908-c4d424ad9a1b\") "
Jul 16 22:43:00.552861 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:00.552832    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3170d5-1a46-4411-9908-c4d424ad9a1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb3170d5-1a46-4411-9908-c4d424ad9a1b" (UID: "eb3170d5-1a46-4411-9908-c4d424ad9a1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:00.553053 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:00.553034    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3170d5-1a46-4411-9908-c4d424ad9a1b-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:43:01.303270 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:01.303239    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"
Jul 16 22:43:01.303742 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:01.303247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n" event={"ID":"eb3170d5-1a46-4411-9908-c4d424ad9a1b","Type":"ContainerDied","Data":"dcc618607ae1ee83d410212a224665d8f983074d606334ab3cd72fca5a4e5f69"}
Jul 16 22:43:01.303742 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:01.303372    2555 scope.go:117] "RemoveContainer" containerID="5d127f8b7f2a63a91b99bb2d357b6c98bf43c782d41ffbab9d53f768302028d6"
Jul 16 22:43:01.326621 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:01.326595    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"]
Jul 16 22:43:01.333409 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:01.333386    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-56h8n"]
Jul 16 22:43:02.494723 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:02.494688    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" path="/var/lib/kubelet/pods/eb3170d5-1a46-4411-9908-c4d424ad9a1b/volumes"
Jul 16 22:43:12.490094 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:12.490059    2555 scope.go:117] "RemoveContainer" containerID="5e7d54ea7a3011ab2d9094b1a726c96ddce997b4b08366c0ed7f031134127184"
Jul 16 22:43:13.362924 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:13.362887    2555 generic.go:356] "Generic (PLEG): container finished" podID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerID="113e0e124dc715c37113c5797affcf5b697e537c5c71af7e2a2c238d9be393ba" exitCode=6
Jul 16 22:43:13.363138 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:13.362936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" event={"ID":"05001318-156f-41de-adc3-6e4dcf9e8ca6","Type":"ContainerDied","Data":"113e0e124dc715c37113c5797affcf5b697e537c5c71af7e2a2c238d9be393ba"}
Jul 16 22:43:13.363138 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:13.363062    2555 scope.go:117] "RemoveContainer" containerID="5e7d54ea7a3011ab2d9094b1a726c96ddce997b4b08366c0ed7f031134127184"
Jul 16 22:43:13.363429 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:13.363414    2555 scope.go:117] "RemoveContainer" containerID="113e0e124dc715c37113c5797affcf5b697e537c5c71af7e2a2c238d9be393ba"
Jul 16 22:43:13.363640 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:13.363613    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx_odh-ai-gateway-infra(05001318-156f-41de-adc3-6e4dcf9e8ca6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6"
Jul 16 22:43:14.396310 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:14.396276    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"]
Jul 16 22:43:14.525110 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:14.525077    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"
Jul 16 22:43:14.581079 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:14.581053    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05001318-156f-41de-adc3-6e4dcf9e8ca6-maas-api-service-ca\") pod \"05001318-156f-41de-adc3-6e4dcf9e8ca6\" (UID: \"05001318-156f-41de-adc3-6e4dcf9e8ca6\") "
Jul 16 22:43:14.581408 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:14.581384    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05001318-156f-41de-adc3-6e4dcf9e8ca6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05001318-156f-41de-adc3-6e4dcf9e8ca6" (UID: "05001318-156f-41de-adc3-6e4dcf9e8ca6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:14.682452 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:14.682368    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05001318-156f-41de-adc3-6e4dcf9e8ca6-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.374869 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:15.374839    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"
Jul 16 22:43:15.374869 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:15.374859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx" event={"ID":"05001318-156f-41de-adc3-6e4dcf9e8ca6","Type":"ContainerDied","Data":"95272055f3573a6c8c057d22da9c33108bf96259eb691d9d1aa8ab08f5495e9c"}
Jul 16 22:43:15.375154 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:15.374913    2555 scope.go:117] "RemoveContainer" containerID="113e0e124dc715c37113c5797affcf5b697e537c5c71af7e2a2c238d9be393ba"
Jul 16 22:43:15.395849 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:15.395824    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"]
Jul 16 22:43:15.397320 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:15.397299    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-952bx"]
Jul 16 22:43:16.495031 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:16.494999    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" path="/var/lib/kubelet/pods/05001318-156f-41de-adc3-6e4dcf9e8ca6/volumes"
Jul 16 22:43:17.750277 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750242    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"]
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750555    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750566    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750581    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750587    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750600    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750606    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750615    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750621    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750628    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:17.750667 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750633    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:17.751004 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750677    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:17.751004 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750686    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.751004 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750691    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.751004 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750697    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05001318-156f-41de-adc3-6e4dcf9e8ca6" containerName="revoke-keys"
Jul 16 22:43:17.751004 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.750702    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:17.755318 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.755296    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"
Jul 16 22:43:17.757978 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.757937    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:43:17.758121 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.758000    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:43:17.762359 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.762322    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"]
Jul 16 22:43:17.808949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.808909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925aed9-2c2e-463d-8d07-d8c260242197-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m\" (UID: \"5925aed9-2c2e-463d-8d07-d8c260242197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"
Jul 16 22:43:17.910100 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.910060    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925aed9-2c2e-463d-8d07-d8c260242197-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m\" (UID: \"5925aed9-2c2e-463d-8d07-d8c260242197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"
Jul 16 22:43:17.910668 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:17.910647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925aed9-2c2e-463d-8d07-d8c260242197-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m\" (UID: \"5925aed9-2c2e-463d-8d07-d8c260242197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"
Jul 16 22:43:18.067352 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:18.067265    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"
Jul 16 22:43:18.207574 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:18.207540    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"]
Jul 16 22:43:18.215142 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:43:18.215107    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5925aed9_2c2e_463d_8d07_d8c260242197.slice/crio-b8001db957ab0f6d24de3d2bfa0c4e24e922a67ad8b079ea2e02bbe5052d0bef WatchSource:0}: Error finding container b8001db957ab0f6d24de3d2bfa0c4e24e922a67ad8b079ea2e02bbe5052d0bef: Status 404 returned error can't find the container with id b8001db957ab0f6d24de3d2bfa0c4e24e922a67ad8b079ea2e02bbe5052d0bef
Jul 16 22:43:18.389367 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:18.389331    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" event={"ID":"5925aed9-2c2e-463d-8d07-d8c260242197","Type":"ContainerStarted","Data":"b8001db957ab0f6d24de3d2bfa0c4e24e922a67ad8b079ea2e02bbe5052d0bef"}
Jul 16 22:43:19.395475 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:19.395442    2555 generic.go:356] "Generic (PLEG): container finished" podID="5925aed9-2c2e-463d-8d07-d8c260242197" containerID="274ae909a8c771193e5769f85a32bb1a9577aa15087422e34166129cfc45cb5e" exitCode=6
Jul 16 22:43:19.395889 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:19.395530    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" event={"ID":"5925aed9-2c2e-463d-8d07-d8c260242197","Type":"ContainerDied","Data":"274ae909a8c771193e5769f85a32bb1a9577aa15087422e34166129cfc45cb5e"}
Jul 16 22:43:19.395889 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:19.395680    2555 scope.go:117] "RemoveContainer" containerID="274ae909a8c771193e5769f85a32bb1a9577aa15087422e34166129cfc45cb5e"
Jul 16 22:43:20.404890 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:20.404856    2555 generic.go:356] "Generic (PLEG): container finished" podID="5925aed9-2c2e-463d-8d07-d8c260242197" containerID="738065eda70327e84421e9cd9689244539b91dfda25e7007e38ab1960c20d6fa" exitCode=6
Jul 16 22:43:20.405401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:20.404942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" event={"ID":"5925aed9-2c2e-463d-8d07-d8c260242197","Type":"ContainerDied","Data":"738065eda70327e84421e9cd9689244539b91dfda25e7007e38ab1960c20d6fa"}
Jul 16 22:43:20.405401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:20.405033    2555 scope.go:117] "RemoveContainer" containerID="274ae909a8c771193e5769f85a32bb1a9577aa15087422e34166129cfc45cb5e"
Jul 16 22:43:20.405401 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:20.405196    2555 scope.go:117] "RemoveContainer" containerID="738065eda70327e84421e9cd9689244539b91dfda25e7007e38ab1960c20d6fa"
Jul 16 22:43:20.405577 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:20.405418    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m_odh-ai-gateway-infra(5925aed9-2c2e-463d-8d07-d8c260242197)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" podUID="5925aed9-2c2e-463d-8d07-d8c260242197"
Jul 16 22:43:21.410969 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:21.410938    2555 scope.go:117] "RemoveContainer" containerID="738065eda70327e84421e9cd9689244539b91dfda25e7007e38ab1960c20d6fa"
Jul 16 22:43:21.411382 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:21.411148    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m_odh-ai-gateway-infra(5925aed9-2c2e-463d-8d07-d8c260242197)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" podUID="5925aed9-2c2e-463d-8d07-d8c260242197"
Jul 16 22:43:32.490009 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:32.489974    2555 scope.go:117] "RemoveContainer" containerID="738065eda70327e84421e9cd9689244539b91dfda25e7007e38ab1960c20d6fa"
Jul 16 22:43:33.463732 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:33.463699    2555 generic.go:356] "Generic (PLEG): container finished" podID="5925aed9-2c2e-463d-8d07-d8c260242197" containerID="d6494923479d3f369ce16ef2a19675a5a077109b2c95481623b67a9a4591b3e8" exitCode=6
Jul 16 22:43:33.463904 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:33.463773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" event={"ID":"5925aed9-2c2e-463d-8d07-d8c260242197","Type":"ContainerDied","Data":"d6494923479d3f369ce16ef2a19675a5a077109b2c95481623b67a9a4591b3e8"}
Jul 16 22:43:33.463904 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:33.463832    2555 scope.go:117] "RemoveContainer" containerID="738065eda70327e84421e9cd9689244539b91dfda25e7007e38ab1960c20d6fa"
Jul 16 22:43:33.464148 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:33.464127    2555 scope.go:117] "RemoveContainer" containerID="d6494923479d3f369ce16ef2a19675a5a077109b2c95481623b67a9a4591b3e8"
Jul 16 22:43:33.464363 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:33.464342    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m_odh-ai-gateway-infra(5925aed9-2c2e-463d-8d07-d8c260242197)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" podUID="5925aed9-2c2e-463d-8d07-d8c260242197"
Jul 16 22:43:34.505190 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:34.505155    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"]
Jul 16 22:43:34.639537 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:34.639508    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"
Jul 16 22:43:34.753237 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:34.753206    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925aed9-2c2e-463d-8d07-d8c260242197-maas-api-service-ca\") pod \"5925aed9-2c2e-463d-8d07-d8c260242197\" (UID: \"5925aed9-2c2e-463d-8d07-d8c260242197\") "
Jul 16 22:43:34.753545 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:34.753522    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5925aed9-2c2e-463d-8d07-d8c260242197-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5925aed9-2c2e-463d-8d07-d8c260242197" (UID: "5925aed9-2c2e-463d-8d07-d8c260242197"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:34.854555 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:34.854521    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5925aed9-2c2e-463d-8d07-d8c260242197-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:43:35.479454 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:35.479421    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"
Jul 16 22:43:35.479454 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:35.479428    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m" event={"ID":"5925aed9-2c2e-463d-8d07-d8c260242197","Type":"ContainerDied","Data":"b8001db957ab0f6d24de3d2bfa0c4e24e922a67ad8b079ea2e02bbe5052d0bef"}
Jul 16 22:43:35.479713 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:35.479523    2555 scope.go:117] "RemoveContainer" containerID="d6494923479d3f369ce16ef2a19675a5a077109b2c95481623b67a9a4591b3e8"
Jul 16 22:43:35.500601 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:35.500565    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"]
Jul 16 22:43:35.506369 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:35.506344    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-gh87m"]
Jul 16 22:43:36.024390 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024360    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"]
Jul 16 22:43:36.024653 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024641    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:36.024701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024655    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:36.024701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024670    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024676    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024683    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024688    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024701    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024887 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024706    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024887 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024755    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024887 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024762    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:36.024887 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.024768    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3170d5-1a46-4411-9908-c4d424ad9a1b" containerName="revoke-keys"
Jul 16 22:43:36.027635 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.027618    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"
Jul 16 22:43:36.030703 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.030679    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:43:36.030854 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.030780    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:43:36.038412 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.038386    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"]
Jul 16 22:43:36.165496 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.165463    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf09cd6-0755-4b6d-9856-8bfd4ec03bae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j\" (UID: \"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"
Jul 16 22:43:36.266512 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.266475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf09cd6-0755-4b6d-9856-8bfd4ec03bae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j\" (UID: \"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"
Jul 16 22:43:36.267118 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.267100    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf09cd6-0755-4b6d-9856-8bfd4ec03bae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j\" (UID: \"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"
Jul 16 22:43:36.339300 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.339267    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"
Jul 16 22:43:36.476012 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.475978    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"]
Jul 16 22:43:36.477801 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:43:36.477770    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf09cd6_0755_4b6d_9856_8bfd4ec03bae.slice/crio-295c53a0b24cae3a1d4b30b88f45ff5b08f5f03fc0bb2d908784a70b17214b64 WatchSource:0}: Error finding container 295c53a0b24cae3a1d4b30b88f45ff5b08f5f03fc0bb2d908784a70b17214b64: Status 404 returned error can't find the container with id 295c53a0b24cae3a1d4b30b88f45ff5b08f5f03fc0bb2d908784a70b17214b64
Jul 16 22:43:36.485820 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.485799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" event={"ID":"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae","Type":"ContainerStarted","Data":"295c53a0b24cae3a1d4b30b88f45ff5b08f5f03fc0bb2d908784a70b17214b64"}
Jul 16 22:43:36.496834 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:36.496804    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" path="/var/lib/kubelet/pods/5925aed9-2c2e-463d-8d07-d8c260242197/volumes"
Jul 16 22:43:37.492161 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:37.492125    2555 generic.go:356] "Generic (PLEG): container finished" podID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerID="8e0ba5f117ac907f1b092bbac16083fab304478b58f4e0640d9e5cb0056b1a0d" exitCode=6
Jul 16 22:43:37.492655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:37.492178    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" event={"ID":"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae","Type":"ContainerDied","Data":"8e0ba5f117ac907f1b092bbac16083fab304478b58f4e0640d9e5cb0056b1a0d"}
Jul 16 22:43:37.492655 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:37.492438    2555 scope.go:117] "RemoveContainer" containerID="8e0ba5f117ac907f1b092bbac16083fab304478b58f4e0640d9e5cb0056b1a0d"
Jul 16 22:43:38.498192 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:38.498158    2555 generic.go:356] "Generic (PLEG): container finished" podID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerID="8fd337e0870ced256c04e6049ab66416c645c0f421d0c6ba88470157d12cf9bc" exitCode=6
Jul 16 22:43:38.498594 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:38.498239    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" event={"ID":"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae","Type":"ContainerDied","Data":"8fd337e0870ced256c04e6049ab66416c645c0f421d0c6ba88470157d12cf9bc"}
Jul 16 22:43:38.498594 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:38.498290    2555 scope.go:117] "RemoveContainer" containerID="8e0ba5f117ac907f1b092bbac16083fab304478b58f4e0640d9e5cb0056b1a0d"
Jul 16 22:43:38.498594 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:38.498418    2555 scope.go:117] "RemoveContainer" containerID="8fd337e0870ced256c04e6049ab66416c645c0f421d0c6ba88470157d12cf9bc"
Jul 16 22:43:38.498739 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:38.498638    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j_odh-ai-gateway-infra(aaf09cd6-0755-4b6d-9856-8bfd4ec03bae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae"
Jul 16 22:43:39.094037 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.094004    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"]
Jul 16 22:43:39.094397 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.094383    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5925aed9-2c2e-463d-8d07-d8c260242197" containerName="revoke-keys"
Jul 16 22:43:39.098445 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.098425    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"
Jul 16 22:43:39.132509 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.132478    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"]
Jul 16 22:43:39.193858 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.193820    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c723b91c-9b94-4d61-8fda-5c76e6cd2752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-ql65h\" (UID: \"c723b91c-9b94-4d61-8fda-5c76e6cd2752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"
Jul 16 22:43:39.295207 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.295165    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c723b91c-9b94-4d61-8fda-5c76e6cd2752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-ql65h\" (UID: \"c723b91c-9b94-4d61-8fda-5c76e6cd2752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"
Jul 16 22:43:39.295790 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.295770    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c723b91c-9b94-4d61-8fda-5c76e6cd2752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-ql65h\" (UID: \"c723b91c-9b94-4d61-8fda-5c76e6cd2752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"
Jul 16 22:43:39.408484 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.408397    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"
Jul 16 22:43:39.504768 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.504741    2555 scope.go:117] "RemoveContainer" containerID="8fd337e0870ced256c04e6049ab66416c645c0f421d0c6ba88470157d12cf9bc"
Jul 16 22:43:39.505157 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:39.504979    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j_odh-ai-gateway-infra(aaf09cd6-0755-4b6d-9856-8bfd4ec03bae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae"
Jul 16 22:43:39.546366 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:39.546119    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"]
Jul 16 22:43:39.548478 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:43:39.548450    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc723b91c_9b94_4d61_8fda_5c76e6cd2752.slice/crio-fa4251f72bd7861c68c1bde8fd19950dc047caff518b200f6121906d07ffc2cc WatchSource:0}: Error finding container fa4251f72bd7861c68c1bde8fd19950dc047caff518b200f6121906d07ffc2cc: Status 404 returned error can't find the container with id fa4251f72bd7861c68c1bde8fd19950dc047caff518b200f6121906d07ffc2cc
Jul 16 22:43:40.509546 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:40.509463    2555 generic.go:356] "Generic (PLEG): container finished" podID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerID="c8fcd15a64d1d56088aba42e7006f6c03710eae47afc417273f5fc64bea7512d" exitCode=6
Jul 16 22:43:40.509925 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:40.509551    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" event={"ID":"c723b91c-9b94-4d61-8fda-5c76e6cd2752","Type":"ContainerDied","Data":"c8fcd15a64d1d56088aba42e7006f6c03710eae47afc417273f5fc64bea7512d"}
Jul 16 22:43:40.509925 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:40.509614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" event={"ID":"c723b91c-9b94-4d61-8fda-5c76e6cd2752","Type":"ContainerStarted","Data":"fa4251f72bd7861c68c1bde8fd19950dc047caff518b200f6121906d07ffc2cc"}
Jul 16 22:43:40.509925 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:40.509748    2555 scope.go:117] "RemoveContainer" containerID="c8fcd15a64d1d56088aba42e7006f6c03710eae47afc417273f5fc64bea7512d"
Jul 16 22:43:41.515654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:41.515553    2555 generic.go:356] "Generic (PLEG): container finished" podID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerID="5eb87215afa13d4a6e32c543769d40895f720345c36400a3c104a349dfc49b9f" exitCode=6
Jul 16 22:43:41.515654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:41.515627    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" event={"ID":"c723b91c-9b94-4d61-8fda-5c76e6cd2752","Type":"ContainerDied","Data":"5eb87215afa13d4a6e32c543769d40895f720345c36400a3c104a349dfc49b9f"}
Jul 16 22:43:41.516136 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:41.515723    2555 scope.go:117] "RemoveContainer" containerID="c8fcd15a64d1d56088aba42e7006f6c03710eae47afc417273f5fc64bea7512d"
Jul 16 22:43:41.516136 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:41.515889    2555 scope.go:117] "RemoveContainer" containerID="5eb87215afa13d4a6e32c543769d40895f720345c36400a3c104a349dfc49b9f"
Jul 16 22:43:41.516236 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:41.516131    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-ql65h_odh-ai-gateway-infra(c723b91c-9b94-4d61-8fda-5c76e6cd2752)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752"
Jul 16 22:43:42.522208 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:42.522179    2555 scope.go:117] "RemoveContainer" containerID="5eb87215afa13d4a6e32c543769d40895f720345c36400a3c104a349dfc49b9f"
Jul 16 22:43:42.522611 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:42.522355    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-ql65h_odh-ai-gateway-infra(c723b91c-9b94-4d61-8fda-5c76e6cd2752)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752"
Jul 16 22:43:46.723843 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:46.723809    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"]
Jul 16 22:43:46.728518 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:46.728499    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"
Jul 16 22:43:46.738037 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:46.738008    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"]
Jul 16 22:43:46.858578 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:46.858523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730a4b70-cea7-467e-afff-e4e56134c8d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m\" (UID: \"730a4b70-cea7-467e-afff-e4e56134c8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"
Jul 16 22:43:46.960092 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:46.960059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730a4b70-cea7-467e-afff-e4e56134c8d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m\" (UID: \"730a4b70-cea7-467e-afff-e4e56134c8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"
Jul 16 22:43:46.960725 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:46.960705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730a4b70-cea7-467e-afff-e4e56134c8d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m\" (UID: \"730a4b70-cea7-467e-afff-e4e56134c8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"
Jul 16 22:43:47.040350 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:47.040243    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"
Jul 16 22:43:47.191927 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:47.191900    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"]
Jul 16 22:43:47.192942 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:43:47.192909    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730a4b70_cea7_467e_afff_e4e56134c8d3.slice/crio-e8b91a14c07badf64010b5b304d0de1f47efb328c18095a35a80db3b23ebb8a5 WatchSource:0}: Error finding container e8b91a14c07badf64010b5b304d0de1f47efb328c18095a35a80db3b23ebb8a5: Status 404 returned error can't find the container with id e8b91a14c07badf64010b5b304d0de1f47efb328c18095a35a80db3b23ebb8a5
Jul 16 22:43:47.545492 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:47.545452    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" event={"ID":"730a4b70-cea7-467e-afff-e4e56134c8d3","Type":"ContainerStarted","Data":"da25b83a5ccca9d32059b21de0c51116b08fd8deaf6de579fdf8a473f8d77b09"}
Jul 16 22:43:47.545710 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:47.545538    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" event={"ID":"730a4b70-cea7-467e-afff-e4e56134c8d3","Type":"ContainerStarted","Data":"e8b91a14c07badf64010b5b304d0de1f47efb328c18095a35a80db3b23ebb8a5"}
Jul 16 22:43:47.586777 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:47.586713    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" podStartSLOduration=1.586694258 podStartE2EDuration="1.586694258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:47.580346259 +0000 UTC m=+3383.573510411" watchObservedRunningTime="2026-07-16 22:43:47.586694258 +0000 UTC m=+3383.579858411"
Jul 16 22:43:48.551894 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:48.551857    2555 generic.go:356] "Generic (PLEG): container finished" podID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerID="da25b83a5ccca9d32059b21de0c51116b08fd8deaf6de579fdf8a473f8d77b09" exitCode=6
Jul 16 22:43:48.552430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:48.551951    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" event={"ID":"730a4b70-cea7-467e-afff-e4e56134c8d3","Type":"ContainerDied","Data":"da25b83a5ccca9d32059b21de0c51116b08fd8deaf6de579fdf8a473f8d77b09"}
Jul 16 22:43:48.552430 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:48.552150    2555 scope.go:117] "RemoveContainer" containerID="da25b83a5ccca9d32059b21de0c51116b08fd8deaf6de579fdf8a473f8d77b09"
Jul 16 22:43:49.558876 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:49.558785    2555 generic.go:356] "Generic (PLEG): container finished" podID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerID="55d6d0d2c6ac031c014ef792a3e3a962216b061c0eb80f4ab36b54e670a58b70" exitCode=6
Jul 16 22:43:49.559289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:49.558871    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" event={"ID":"730a4b70-cea7-467e-afff-e4e56134c8d3","Type":"ContainerDied","Data":"55d6d0d2c6ac031c014ef792a3e3a962216b061c0eb80f4ab36b54e670a58b70"}
Jul 16 22:43:49.559289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:49.558926    2555 scope.go:117] "RemoveContainer" containerID="da25b83a5ccca9d32059b21de0c51116b08fd8deaf6de579fdf8a473f8d77b09"
Jul 16 22:43:49.559289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:49.559152    2555 scope.go:117] "RemoveContainer" containerID="55d6d0d2c6ac031c014ef792a3e3a962216b061c0eb80f4ab36b54e670a58b70"
Jul 16 22:43:49.559407 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:49.559361    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m_odh-ai-gateway-infra(730a4b70-cea7-467e-afff-e4e56134c8d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3"
Jul 16 22:43:50.565118 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:50.565087    2555 scope.go:117] "RemoveContainer" containerID="55d6d0d2c6ac031c014ef792a3e3a962216b061c0eb80f4ab36b54e670a58b70"
Jul 16 22:43:50.565507 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:50.565297    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m_odh-ai-gateway-infra(730a4b70-cea7-467e-afff-e4e56134c8d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3"
Jul 16 22:43:53.489792 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:53.489761    2555 scope.go:117] "RemoveContainer" containerID="5eb87215afa13d4a6e32c543769d40895f720345c36400a3c104a349dfc49b9f"
Jul 16 22:43:54.214513 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.214478    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"]
Jul 16 22:43:54.217755 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.217732    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"
Jul 16 22:43:54.232886 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.232857    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"]
Jul 16 22:43:54.322585 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.322476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72451d71-b76e-4bd9-bad6-50e26726d0b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw\" (UID: \"72451d71-b76e-4bd9-bad6-50e26726d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"
Jul 16 22:43:54.422975 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.422928    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72451d71-b76e-4bd9-bad6-50e26726d0b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw\" (UID: \"72451d71-b76e-4bd9-bad6-50e26726d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"
Jul 16 22:43:54.423704 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.423683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72451d71-b76e-4bd9-bad6-50e26726d0b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw\" (UID: \"72451d71-b76e-4bd9-bad6-50e26726d0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"
Jul 16 22:43:54.491654 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.491620    2555 scope.go:117] "RemoveContainer" containerID="8fd337e0870ced256c04e6049ab66416c645c0f421d0c6ba88470157d12cf9bc"
Jul 16 22:43:54.526949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.526920    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"
Jul 16 22:43:54.587432 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.587402    2555 generic.go:356] "Generic (PLEG): container finished" podID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerID="e7f1d8a152668c203d9ddd0b9b3a7ff8238e47033ecb933fe37079b3fb5579a0" exitCode=6
Jul 16 22:43:54.587569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.587484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" event={"ID":"c723b91c-9b94-4d61-8fda-5c76e6cd2752","Type":"ContainerDied","Data":"e7f1d8a152668c203d9ddd0b9b3a7ff8238e47033ecb933fe37079b3fb5579a0"}
Jul 16 22:43:54.587569 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.587547    2555 scope.go:117] "RemoveContainer" containerID="5eb87215afa13d4a6e32c543769d40895f720345c36400a3c104a349dfc49b9f"
Jul 16 22:43:54.588299 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.587944    2555 scope.go:117] "RemoveContainer" containerID="e7f1d8a152668c203d9ddd0b9b3a7ff8238e47033ecb933fe37079b3fb5579a0"
Jul 16 22:43:54.588299 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:54.588231    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-ql65h_odh-ai-gateway-infra(c723b91c-9b94-4d61-8fda-5c76e6cd2752)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752"
Jul 16 22:43:54.669203 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:54.669177    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"]
Jul 16 22:43:54.670323 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:43:54.670294    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72451d71_b76e_4bd9_bad6_50e26726d0b9.slice/crio-1246a6ddf48f51d982878e0ea9f67d883a0d0def5151c683ae6f9aba97ca3b5f WatchSource:0}: Error finding container 1246a6ddf48f51d982878e0ea9f67d883a0d0def5151c683ae6f9aba97ca3b5f: Status 404 returned error can't find the container with id 1246a6ddf48f51d982878e0ea9f67d883a0d0def5151c683ae6f9aba97ca3b5f
Jul 16 22:43:55.593866 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.593829    2555 generic.go:356] "Generic (PLEG): container finished" podID="72451d71-b76e-4bd9-bad6-50e26726d0b9" containerID="c4c27a5c1e442bfbb6198e75c9696839f903880dcf47489753514ff8245d7c53" exitCode=0
Jul 16 22:43:55.594339 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.593905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw" event={"ID":"72451d71-b76e-4bd9-bad6-50e26726d0b9","Type":"ContainerDied","Data":"c4c27a5c1e442bfbb6198e75c9696839f903880dcf47489753514ff8245d7c53"}
Jul 16 22:43:55.594339 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.593965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw" event={"ID":"72451d71-b76e-4bd9-bad6-50e26726d0b9","Type":"ContainerStarted","Data":"1246a6ddf48f51d982878e0ea9f67d883a0d0def5151c683ae6f9aba97ca3b5f"}
Jul 16 22:43:55.596947 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.596926    2555 generic.go:356] "Generic (PLEG): container finished" podID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerID="cbde5e29f98c03ea9f811b4fd0e40c434fc9a560e876d64d771ef3e75da4ee22" exitCode=6
Jul 16 22:43:55.597077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.596988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" event={"ID":"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae","Type":"ContainerDied","Data":"cbde5e29f98c03ea9f811b4fd0e40c434fc9a560e876d64d771ef3e75da4ee22"}
Jul 16 22:43:55.597077 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.597032    2555 scope.go:117] "RemoveContainer" containerID="8fd337e0870ced256c04e6049ab66416c645c0f421d0c6ba88470157d12cf9bc"
Jul 16 22:43:55.597299 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.597283    2555 scope.go:117] "RemoveContainer" containerID="cbde5e29f98c03ea9f811b4fd0e40c434fc9a560e876d64d771ef3e75da4ee22"
Jul 16 22:43:55.597560 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:43:55.597528    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j_odh-ai-gateway-infra(aaf09cd6-0755-4b6d-9856-8bfd4ec03bae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae"
Jul 16 22:43:55.627413 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.627390    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"]
Jul 16 22:43:55.762547 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.762484    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"
Jul 16 22:43:55.935444 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.935411    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c723b91c-9b94-4d61-8fda-5c76e6cd2752-maas-api-service-ca\") pod \"c723b91c-9b94-4d61-8fda-5c76e6cd2752\" (UID: \"c723b91c-9b94-4d61-8fda-5c76e6cd2752\") "
Jul 16 22:43:55.935753 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:55.935732    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c723b91c-9b94-4d61-8fda-5c76e6cd2752-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c723b91c-9b94-4d61-8fda-5c76e6cd2752" (UID: "c723b91c-9b94-4d61-8fda-5c76e6cd2752"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:56.036767 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.036683    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c723b91c-9b94-4d61-8fda-5c76e6cd2752-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:43:56.603127 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.603096    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"
Jul 16 22:43:56.603588 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.603135    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h" event={"ID":"c723b91c-9b94-4d61-8fda-5c76e6cd2752","Type":"ContainerDied","Data":"fa4251f72bd7861c68c1bde8fd19950dc047caff518b200f6121906d07ffc2cc"}
Jul 16 22:43:56.603588 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.603209    2555 scope.go:117] "RemoveContainer" containerID="e7f1d8a152668c203d9ddd0b9b3a7ff8238e47033ecb933fe37079b3fb5579a0"
Jul 16 22:43:56.636309 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.636274    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"]
Jul 16 22:43:56.638486 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.638454    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-ql65h"]
Jul 16 22:43:56.643101 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.643077    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"]
Jul 16 22:43:56.740736 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.740707    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"
Jul 16 22:43:56.792078 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.792053    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"
Jul 16 22:43:56.844118 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.844078    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72451d71-b76e-4bd9-bad6-50e26726d0b9-maas-api-service-ca\") pod \"72451d71-b76e-4bd9-bad6-50e26726d0b9\" (UID: \"72451d71-b76e-4bd9-bad6-50e26726d0b9\") "
Jul 16 22:43:56.844435 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.844412    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72451d71-b76e-4bd9-bad6-50e26726d0b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72451d71-b76e-4bd9-bad6-50e26726d0b9" (UID: "72451d71-b76e-4bd9-bad6-50e26726d0b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:56.945019 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.944940    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf09cd6-0755-4b6d-9856-8bfd4ec03bae-maas-api-service-ca\") pod \"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae\" (UID: \"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae\") "
Jul 16 22:43:56.945206 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.945189    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72451d71-b76e-4bd9-bad6-50e26726d0b9-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:43:56.945314 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:56.945290    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf09cd6-0755-4b6d-9856-8bfd4ec03bae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" (UID: "aaf09cd6-0755-4b6d-9856-8bfd4ec03bae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:57.045874 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.045785    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf09cd6-0755-4b6d-9856-8bfd4ec03bae-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:43:57.610533 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.610502    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"
Jul 16 22:43:57.610533 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.610522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j" event={"ID":"aaf09cd6-0755-4b6d-9856-8bfd4ec03bae","Type":"ContainerDied","Data":"295c53a0b24cae3a1d4b30b88f45ff5b08f5f03fc0bb2d908784a70b17214b64"}
Jul 16 22:43:57.611098 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.610583    2555 scope.go:117] "RemoveContainer" containerID="cbde5e29f98c03ea9f811b4fd0e40c434fc9a560e876d64d771ef3e75da4ee22"
Jul 16 22:43:57.611902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.611726    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw"
Jul 16 22:43:57.611902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.611719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw" event={"ID":"72451d71-b76e-4bd9-bad6-50e26726d0b9","Type":"ContainerDied","Data":"1246a6ddf48f51d982878e0ea9f67d883a0d0def5151c683ae6f9aba97ca3b5f"}
Jul 16 22:43:57.611902 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.611848    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1246a6ddf48f51d982878e0ea9f67d883a0d0def5151c683ae6f9aba97ca3b5f"
Jul 16 22:43:57.637618 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.637592    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"]
Jul 16 22:43:57.639230 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:57.639210    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hvr4j"]
Jul 16 22:43:58.495095 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:58.495062    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" path="/var/lib/kubelet/pods/aaf09cd6-0755-4b6d-9856-8bfd4ec03bae/volumes"
Jul 16 22:43:58.495407 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:43:58.495394    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" path="/var/lib/kubelet/pods/c723b91c-9b94-4d61-8fda-5c76e6cd2752/volumes"
Jul 16 22:44:02.495831 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:02.495802    2555 scope.go:117] "RemoveContainer" containerID="55d6d0d2c6ac031c014ef792a3e3a962216b061c0eb80f4ab36b54e670a58b70"
Jul 16 22:44:03.641782 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:03.641748    2555 generic.go:356] "Generic (PLEG): container finished" podID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerID="1fd2556b75dfea925f348150712e593e9c1526ea52e634142c8afb7d32f65083" exitCode=6
Jul 16 22:44:03.642215 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:03.641809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" event={"ID":"730a4b70-cea7-467e-afff-e4e56134c8d3","Type":"ContainerDied","Data":"1fd2556b75dfea925f348150712e593e9c1526ea52e634142c8afb7d32f65083"}
Jul 16 22:44:03.642215 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:03.641865    2555 scope.go:117] "RemoveContainer" containerID="55d6d0d2c6ac031c014ef792a3e3a962216b061c0eb80f4ab36b54e670a58b70"
Jul 16 22:44:03.642215 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:03.642159    2555 scope.go:117] "RemoveContainer" containerID="1fd2556b75dfea925f348150712e593e9c1526ea52e634142c8afb7d32f65083"
Jul 16 22:44:03.642414 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:03.642393    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m_odh-ai-gateway-infra(730a4b70-cea7-467e-afff-e4e56134c8d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3"
Jul 16 22:44:04.679591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:04.679552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"]
Jul 16 22:44:04.825083 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:04.825057    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"
Jul 16 22:44:04.916145 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:04.916108    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730a4b70-cea7-467e-afff-e4e56134c8d3-maas-api-service-ca\") pod \"730a4b70-cea7-467e-afff-e4e56134c8d3\" (UID: \"730a4b70-cea7-467e-afff-e4e56134c8d3\") "
Jul 16 22:44:04.916463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:04.916442    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730a4b70-cea7-467e-afff-e4e56134c8d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "730a4b70-cea7-467e-afff-e4e56134c8d3" (UID: "730a4b70-cea7-467e-afff-e4e56134c8d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:05.017225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:05.017140    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/730a4b70-cea7-467e-afff-e4e56134c8d3-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:44:05.652919 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:05.652881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m" event={"ID":"730a4b70-cea7-467e-afff-e4e56134c8d3","Type":"ContainerDied","Data":"e8b91a14c07badf64010b5b304d0de1f47efb328c18095a35a80db3b23ebb8a5"}
Jul 16 22:44:05.653153 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:05.652947    2555 scope.go:117] "RemoveContainer" containerID="1fd2556b75dfea925f348150712e593e9c1526ea52e634142c8afb7d32f65083"
Jul 16 22:44:05.653153 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:05.652889    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"
Jul 16 22:44:05.683520 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:05.683488    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"]
Jul 16 22:44:05.688838 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:05.688803    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-vjb6m"]
Jul 16 22:44:06.498335 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:06.498300    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" path="/var/lib/kubelet/pods/730a4b70-cea7-467e-afff-e4e56134c8d3/volumes"
Jul 16 22:44:07.827586 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827548    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"]
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827843    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827854    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827864    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827869    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827875    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827880    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827888    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827894    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827901    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827906    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827912    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827917    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827922    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827927    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827933    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72451d71-b76e-4bd9-bad6-50e26726d0b9" containerName="revoke-keys"
Jul 16 22:44:07.828000 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.827937    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="72451d71-b76e-4bd9-bad6-50e26726d0b9" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828019    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="72451d71-b76e-4bd9-bad6-50e26726d0b9" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828029    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828035    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828040    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828045    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828051    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c723b91c-9b94-4d61-8fda-5c76e6cd2752" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828056    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828061    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:07.828473 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.828066    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:07.830936 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.830920    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"
Jul 16 22:44:07.834261 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.834243    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:44:07.834719 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.834702    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:44:07.843640 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.843611    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f93b9d-1d90-45e9-ab85-40ea045bfdd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8\" (UID: \"67f93b9d-1d90-45e9-ab85-40ea045bfdd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"
Jul 16 22:44:07.843936 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.843908    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"]
Jul 16 22:44:07.944586 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.944542    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f93b9d-1d90-45e9-ab85-40ea045bfdd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8\" (UID: \"67f93b9d-1d90-45e9-ab85-40ea045bfdd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"
Jul 16 22:44:07.945187 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:07.945169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f93b9d-1d90-45e9-ab85-40ea045bfdd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8\" (UID: \"67f93b9d-1d90-45e9-ab85-40ea045bfdd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"
Jul 16 22:44:08.141505 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:08.141414    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"
Jul 16 22:44:08.304183 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:08.304152    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"]
Jul 16 22:44:08.307474 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:44:08.307450    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f93b9d_1d90_45e9_ab85_40ea045bfdd4.slice/crio-f3363aace0e48d9a0f0f3b052e93e409cec295ec4850d2627ea2b98d56311037 WatchSource:0}: Error finding container f3363aace0e48d9a0f0f3b052e93e409cec295ec4850d2627ea2b98d56311037: Status 404 returned error can't find the container with id f3363aace0e48d9a0f0f3b052e93e409cec295ec4850d2627ea2b98d56311037
Jul 16 22:44:08.668160 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:08.668125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" event={"ID":"67f93b9d-1d90-45e9-ab85-40ea045bfdd4","Type":"ContainerStarted","Data":"dc5e1004fbf7795b3ef017032f8c596edd68232e87ee2c2331f9f437cc3b336b"}
Jul 16 22:44:08.668300 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:08.668176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" event={"ID":"67f93b9d-1d90-45e9-ab85-40ea045bfdd4","Type":"ContainerStarted","Data":"f3363aace0e48d9a0f0f3b052e93e409cec295ec4850d2627ea2b98d56311037"}
Jul 16 22:44:08.690563 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:08.690505    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" podStartSLOduration=1.690488389 podStartE2EDuration="1.690488389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:08.688305733 +0000 UTC m=+3404.681469910" watchObservedRunningTime="2026-07-16 22:44:08.690488389 +0000 UTC m=+3404.683652541"
Jul 16 22:44:09.673971 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:09.673918    2555 generic.go:356] "Generic (PLEG): container finished" podID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerID="dc5e1004fbf7795b3ef017032f8c596edd68232e87ee2c2331f9f437cc3b336b" exitCode=6
Jul 16 22:44:09.674432 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:09.674008    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" event={"ID":"67f93b9d-1d90-45e9-ab85-40ea045bfdd4","Type":"ContainerDied","Data":"dc5e1004fbf7795b3ef017032f8c596edd68232e87ee2c2331f9f437cc3b336b"}
Jul 16 22:44:09.674432 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:09.674192    2555 scope.go:117] "RemoveContainer" containerID="dc5e1004fbf7795b3ef017032f8c596edd68232e87ee2c2331f9f437cc3b336b"
Jul 16 22:44:10.680327 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:10.680294    2555 generic.go:356] "Generic (PLEG): container finished" podID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerID="7bf071601b58c1aa50d9653f8a96476cd0b4a12ce2d37a82f70490d1c9d1124b" exitCode=6
Jul 16 22:44:10.680797 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:10.680350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" event={"ID":"67f93b9d-1d90-45e9-ab85-40ea045bfdd4","Type":"ContainerDied","Data":"7bf071601b58c1aa50d9653f8a96476cd0b4a12ce2d37a82f70490d1c9d1124b"}
Jul 16 22:44:10.680797 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:10.680398    2555 scope.go:117] "RemoveContainer" containerID="dc5e1004fbf7795b3ef017032f8c596edd68232e87ee2c2331f9f437cc3b336b"
Jul 16 22:44:10.680797 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:10.680579    2555 scope.go:117] "RemoveContainer" containerID="7bf071601b58c1aa50d9653f8a96476cd0b4a12ce2d37a82f70490d1c9d1124b"
Jul 16 22:44:10.681017 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:10.680814    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8_odh-ai-gateway-infra(67f93b9d-1d90-45e9-ab85-40ea045bfdd4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4"
Jul 16 22:44:11.686289 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:11.686261    2555 scope.go:117] "RemoveContainer" containerID="7bf071601b58c1aa50d9653f8a96476cd0b4a12ce2d37a82f70490d1c9d1124b"
Jul 16 22:44:11.686679 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:11.686423    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8_odh-ai-gateway-infra(67f93b9d-1d90-45e9-ab85-40ea045bfdd4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4"
Jul 16 22:44:20.686787 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.686707    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"]
Jul 16 22:44:20.687361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.687165    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:20.687361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.687185    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:20.687361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.687199    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:20.687361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.687209    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf09cd6-0755-4b6d-9856-8bfd4ec03bae" containerName="revoke-keys"
Jul 16 22:44:20.687361 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.687310    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="730a4b70-cea7-467e-afff-e4e56134c8d3" containerName="revoke-keys"
Jul 16 22:44:20.691619 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.691598    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"
Jul 16 22:44:20.709949 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.709927    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"]
Jul 16 22:44:20.759267 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.759230    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf07e1-f639-408a-bc8c-faba0eadb88d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf\" (UID: \"f0bf07e1-f639-408a-bc8c-faba0eadb88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"
Jul 16 22:44:20.860674 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.860638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf07e1-f639-408a-bc8c-faba0eadb88d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf\" (UID: \"f0bf07e1-f639-408a-bc8c-faba0eadb88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"
Jul 16 22:44:20.861259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:20.861239    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf07e1-f639-408a-bc8c-faba0eadb88d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf\" (UID: \"f0bf07e1-f639-408a-bc8c-faba0eadb88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"
Jul 16 22:44:21.006106 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:21.006001    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"
Jul 16 22:44:21.156011 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:21.155984    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"]
Jul 16 22:44:21.158306 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:44:21.158274    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bf07e1_f639_408a_bc8c_faba0eadb88d.slice/crio-02da68e8b05972474e02fe40f6a1ee6ff6a6084b523779f0460dcbb937df5283 WatchSource:0}: Error finding container 02da68e8b05972474e02fe40f6a1ee6ff6a6084b523779f0460dcbb937df5283: Status 404 returned error can't find the container with id 02da68e8b05972474e02fe40f6a1ee6ff6a6084b523779f0460dcbb937df5283
Jul 16 22:44:21.736809 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:21.736713    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf" event={"ID":"f0bf07e1-f639-408a-bc8c-faba0eadb88d","Type":"ContainerStarted","Data":"b557db39c04624a2e556267a01a1191b5522c0e803011962045ca94f848df816"}
Jul 16 22:44:21.736809 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:21.736772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf" event={"ID":"f0bf07e1-f639-408a-bc8c-faba0eadb88d","Type":"ContainerStarted","Data":"02da68e8b05972474e02fe40f6a1ee6ff6a6084b523779f0460dcbb937df5283"}
Jul 16 22:44:21.772402 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:21.772350    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf" podStartSLOduration=1.772336035 podStartE2EDuration="1.772336035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:21.770867588 +0000 UTC m=+3417.764031741" watchObservedRunningTime="2026-07-16 22:44:21.772336035 +0000 UTC m=+3417.765500194"
Jul 16 22:44:22.742303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:22.742271    2555 generic.go:356] "Generic (PLEG): container finished" podID="f0bf07e1-f639-408a-bc8c-faba0eadb88d" containerID="b557db39c04624a2e556267a01a1191b5522c0e803011962045ca94f848df816" exitCode=0
Jul 16 22:44:22.742674 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:22.742356    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf" event={"ID":"f0bf07e1-f639-408a-bc8c-faba0eadb88d","Type":"ContainerDied","Data":"b557db39c04624a2e556267a01a1191b5522c0e803011962045ca94f848df816"}
Jul 16 22:44:23.875002 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:23.874977    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"
Jul 16 22:44:23.988779 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:23.988739    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf07e1-f639-408a-bc8c-faba0eadb88d-maas-api-service-ca\") pod \"f0bf07e1-f639-408a-bc8c-faba0eadb88d\" (UID: \"f0bf07e1-f639-408a-bc8c-faba0eadb88d\") "
Jul 16 22:44:23.989124 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:23.989100    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bf07e1-f639-408a-bc8c-faba0eadb88d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0bf07e1-f639-408a-bc8c-faba0eadb88d" (UID: "f0bf07e1-f639-408a-bc8c-faba0eadb88d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:24.090170 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:24.090129    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf07e1-f639-408a-bc8c-faba0eadb88d-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:44:24.491406 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:24.491321    2555 scope.go:117] "RemoveContainer" containerID="7bf071601b58c1aa50d9653f8a96476cd0b4a12ce2d37a82f70490d1c9d1124b"
Jul 16 22:44:24.753497 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:24.753412    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" event={"ID":"67f93b9d-1d90-45e9-ab85-40ea045bfdd4","Type":"ContainerStarted","Data":"7e980e32c0df830bbb4d9ee6f5dc6e98c949e1aab9e115dca410dcb0d1b8f5b0"}
Jul 16 22:44:24.754652 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:24.754628    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf" event={"ID":"f0bf07e1-f639-408a-bc8c-faba0eadb88d","Type":"ContainerDied","Data":"02da68e8b05972474e02fe40f6a1ee6ff6a6084b523779f0460dcbb937df5283"}
Jul 16 22:44:24.754762 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:24.754669    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02da68e8b05972474e02fe40f6a1ee6ff6a6084b523779f0460dcbb937df5283"
Jul 16 22:44:24.754762 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:24.754636    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf"
Jul 16 22:44:25.760576 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:25.760537    2555 generic.go:356] "Generic (PLEG): container finished" podID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerID="7e980e32c0df830bbb4d9ee6f5dc6e98c949e1aab9e115dca410dcb0d1b8f5b0" exitCode=6
Jul 16 22:44:25.761059 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:25.760613    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" event={"ID":"67f93b9d-1d90-45e9-ab85-40ea045bfdd4","Type":"ContainerDied","Data":"7e980e32c0df830bbb4d9ee6f5dc6e98c949e1aab9e115dca410dcb0d1b8f5b0"}
Jul 16 22:44:25.761059 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:25.760692    2555 scope.go:117] "RemoveContainer" containerID="7bf071601b58c1aa50d9653f8a96476cd0b4a12ce2d37a82f70490d1c9d1124b"
Jul 16 22:44:25.761059 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:25.760981    2555 scope.go:117] "RemoveContainer" containerID="7e980e32c0df830bbb4d9ee6f5dc6e98c949e1aab9e115dca410dcb0d1b8f5b0"
Jul 16 22:44:25.761217 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:25.761173    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8_odh-ai-gateway-infra(67f93b9d-1d90-45e9-ab85-40ea045bfdd4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4"
Jul 16 22:44:25.787409 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:25.787384    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"]
Jul 16 22:44:26.894269 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:26.894244    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"
Jul 16 22:44:27.015807 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.015773    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f93b9d-1d90-45e9-ab85-40ea045bfdd4-maas-api-service-ca\") pod \"67f93b9d-1d90-45e9-ab85-40ea045bfdd4\" (UID: \"67f93b9d-1d90-45e9-ab85-40ea045bfdd4\") "
Jul 16 22:44:27.016138 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.016115    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f93b9d-1d90-45e9-ab85-40ea045bfdd4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67f93b9d-1d90-45e9-ab85-40ea045bfdd4" (UID: "67f93b9d-1d90-45e9-ab85-40ea045bfdd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:27.117046 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.117009    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f93b9d-1d90-45e9-ab85-40ea045bfdd4-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:44:27.772757 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.772719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8" event={"ID":"67f93b9d-1d90-45e9-ab85-40ea045bfdd4","Type":"ContainerDied","Data":"f3363aace0e48d9a0f0f3b052e93e409cec295ec4850d2627ea2b98d56311037"}
Jul 16 22:44:27.772757 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.772750    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"
Jul 16 22:44:27.773029 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.772780    2555 scope.go:117] "RemoveContainer" containerID="7e980e32c0df830bbb4d9ee6f5dc6e98c949e1aab9e115dca410dcb0d1b8f5b0"
Jul 16 22:44:27.798074 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.798048    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"]
Jul 16 22:44:27.808935 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:27.808912    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-zpwr8"]
Jul 16 22:44:28.495490 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:28.495457    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" path="/var/lib/kubelet/pods/67f93b9d-1d90-45e9-ab85-40ea045bfdd4/volumes"
Jul 16 22:44:29.180713 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.180670    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"]
Jul 16 22:44:29.181071 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181053    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:44:29.181163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181074    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:44:29.181163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181101    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:44:29.181163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181111    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:44:29.181163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181120    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bf07e1-f639-408a-bc8c-faba0eadb88d" containerName="revoke-keys"
Jul 16 22:44:29.181163 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181129    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bf07e1-f639-408a-bc8c-faba0eadb88d" containerName="revoke-keys"
Jul 16 22:44:29.181426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181224    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:44:29.181426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181239    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:44:29.181426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181247    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:44:29.181426 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.181258    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bf07e1-f639-408a-bc8c-faba0eadb88d" containerName="revoke-keys"
Jul 16 22:44:29.185749 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.185727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"
Jul 16 22:44:29.189006 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.188986    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:44:29.189107 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.189035    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:44:29.194795 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.194769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"]
Jul 16 22:44:29.336185 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.336131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac791d1-ea04-4a2e-8541-7fd541e0d9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-7rd59\" (UID: \"bac791d1-ea04-4a2e-8541-7fd541e0d9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"
Jul 16 22:44:29.437141 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.437054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac791d1-ea04-4a2e-8541-7fd541e0d9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-7rd59\" (UID: \"bac791d1-ea04-4a2e-8541-7fd541e0d9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"
Jul 16 22:44:29.437730 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.437709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac791d1-ea04-4a2e-8541-7fd541e0d9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-7rd59\" (UID: \"bac791d1-ea04-4a2e-8541-7fd541e0d9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"
Jul 16 22:44:29.496990 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.496941    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"
Jul 16 22:44:29.622180 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.622156    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"]
Jul 16 22:44:29.623901 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:44:29.623876    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac791d1_ea04_4a2e_8541_7fd541e0d9ab.slice/crio-d0261f7e25030a348a986f9e719fa0aaf68e4dc88c6fb26a311ef29a7b92c626 WatchSource:0}: Error finding container d0261f7e25030a348a986f9e719fa0aaf68e4dc88c6fb26a311ef29a7b92c626: Status 404 returned error can't find the container with id d0261f7e25030a348a986f9e719fa0aaf68e4dc88c6fb26a311ef29a7b92c626
Jul 16 22:44:29.782503 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:29.782470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" event={"ID":"bac791d1-ea04-4a2e-8541-7fd541e0d9ab","Type":"ContainerStarted","Data":"d0261f7e25030a348a986f9e719fa0aaf68e4dc88c6fb26a311ef29a7b92c626"}
Jul 16 22:44:30.272474 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:30.272443    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac791d1_ea04_4a2e_8541_7fd541e0d9ab.slice/crio-8c5525c2c7edfc1e2f8c395e6c692982b77bdee43d43b29b5d25eab3db27b428.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:44:30.788673 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:30.788636    2555 generic.go:356] "Generic (PLEG): container finished" podID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerID="8c5525c2c7edfc1e2f8c395e6c692982b77bdee43d43b29b5d25eab3db27b428" exitCode=6
Jul 16 22:44:30.789089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:30.788721    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" event={"ID":"bac791d1-ea04-4a2e-8541-7fd541e0d9ab","Type":"ContainerDied","Data":"8c5525c2c7edfc1e2f8c395e6c692982b77bdee43d43b29b5d25eab3db27b428"}
Jul 16 22:44:30.789089 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:30.788903    2555 scope.go:117] "RemoveContainer" containerID="8c5525c2c7edfc1e2f8c395e6c692982b77bdee43d43b29b5d25eab3db27b428"
Jul 16 22:44:31.794645 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:31.794557    2555 generic.go:356] "Generic (PLEG): container finished" podID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerID="6097f55369add8b431e7c37abb35dd74e869d35e81b494bdf1891600a014d268" exitCode=6
Jul 16 22:44:31.795090 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:31.794644    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" event={"ID":"bac791d1-ea04-4a2e-8541-7fd541e0d9ab","Type":"ContainerDied","Data":"6097f55369add8b431e7c37abb35dd74e869d35e81b494bdf1891600a014d268"}
Jul 16 22:44:31.795090 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:31.794712    2555 scope.go:117] "RemoveContainer" containerID="8c5525c2c7edfc1e2f8c395e6c692982b77bdee43d43b29b5d25eab3db27b428"
Jul 16 22:44:31.795090 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:31.794903    2555 scope.go:117] "RemoveContainer" containerID="6097f55369add8b431e7c37abb35dd74e869d35e81b494bdf1891600a014d268"
Jul 16 22:44:31.795230 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:31.795152    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-7rd59_odh-ai-gateway-infra(bac791d1-ea04-4a2e-8541-7fd541e0d9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab"
Jul 16 22:44:32.801227 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:32.801190    2555 scope.go:117] "RemoveContainer" containerID="6097f55369add8b431e7c37abb35dd74e869d35e81b494bdf1891600a014d268"
Jul 16 22:44:32.801617 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:32.801366    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-7rd59_odh-ai-gateway-infra(bac791d1-ea04-4a2e-8541-7fd541e0d9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab"
Jul 16 22:44:46.495090 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:46.495059    2555 scope.go:117] "RemoveContainer" containerID="6097f55369add8b431e7c37abb35dd74e869d35e81b494bdf1891600a014d268"
Jul 16 22:44:46.869788 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:46.869754    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" event={"ID":"bac791d1-ea04-4a2e-8541-7fd541e0d9ab","Type":"ContainerStarted","Data":"43fbd36c592dda22015b6b13718f83019f311c06faf6887bd0bd24a880b49fbb"}
Jul 16 22:44:47.877279 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:47.877247    2555 generic.go:356] "Generic (PLEG): container finished" podID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerID="43fbd36c592dda22015b6b13718f83019f311c06faf6887bd0bd24a880b49fbb" exitCode=6
Jul 16 22:44:47.877766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:47.877299    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" event={"ID":"bac791d1-ea04-4a2e-8541-7fd541e0d9ab","Type":"ContainerDied","Data":"43fbd36c592dda22015b6b13718f83019f311c06faf6887bd0bd24a880b49fbb"}
Jul 16 22:44:47.877766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:47.877346    2555 scope.go:117] "RemoveContainer" containerID="6097f55369add8b431e7c37abb35dd74e869d35e81b494bdf1891600a014d268"
Jul 16 22:44:47.877766 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:47.877688    2555 scope.go:117] "RemoveContainer" containerID="43fbd36c592dda22015b6b13718f83019f311c06faf6887bd0bd24a880b49fbb"
Jul 16 22:44:47.877913 ip-10-0-143-124 kubenswrapper[2555]: E0716 22:44:47.877889    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-7rd59_odh-ai-gateway-infra(bac791d1-ea04-4a2e-8541-7fd541e0d9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab"
Jul 16 22:44:48.054631 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:48.054593    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"]
Jul 16 22:44:49.007303 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.007277    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"
Jul 16 22:44:49.016207 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.016184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac791d1-ea04-4a2e-8541-7fd541e0d9ab-maas-api-service-ca\") pod \"bac791d1-ea04-4a2e-8541-7fd541e0d9ab\" (UID: \"bac791d1-ea04-4a2e-8541-7fd541e0d9ab\") "
Jul 16 22:44:49.016549 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.016528    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac791d1-ea04-4a2e-8541-7fd541e0d9ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bac791d1-ea04-4a2e-8541-7fd541e0d9ab" (UID: "bac791d1-ea04-4a2e-8541-7fd541e0d9ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:49.116752 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.116717    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac791d1-ea04-4a2e-8541-7fd541e0d9ab-maas-api-service-ca\") on node \"ip-10-0-143-124.ec2.internal\" DevicePath \"\""
Jul 16 22:44:49.888707 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.888674    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"
Jul 16 22:44:49.890842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.888674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59" event={"ID":"bac791d1-ea04-4a2e-8541-7fd541e0d9ab","Type":"ContainerDied","Data":"d0261f7e25030a348a986f9e719fa0aaf68e4dc88c6fb26a311ef29a7b92c626"}
Jul 16 22:44:49.890842 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.888810    2555 scope.go:117] "RemoveContainer" containerID="43fbd36c592dda22015b6b13718f83019f311c06faf6887bd0bd24a880b49fbb"
Jul 16 22:44:49.918769 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.918736    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"]
Jul 16 22:44:49.921989 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:49.921951    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-7rd59"]
Jul 16 22:44:50.500701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:50.500663    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" path="/var/lib/kubelet/pods/bac791d1-ea04-4a2e-8541-7fd541e0d9ab/volumes"
Jul 16 22:44:54.034233 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:54.034205    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-77cfc_3c33481c-6a26-4613-98f7-df5125cf3b05/copy-manifests/0.log"
Jul 16 22:44:54.054651 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:54.054607    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-77cfc_3c33481c-6a26-4613-98f7-df5125cf3b05/manager/0.log"
Jul 16 22:44:55.097993 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.097943    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-8247b5-hrsl5_72801518-24a8-4126-8b4d-6665ae3dc9c8/revoke-keys/0.log"
Jul 16 22:44:55.217497 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.217471    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-1f2f96-wv82s_f2302143-f76f-4432-9cad-2aa49fd25f68/revoke-keys/0.log"
Jul 16 22:44:55.340185 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.340156    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-c36d62-4p5v7_7605ccb9-5558-4bb8-86f9-a837f73ebf6f/revoke-keys/0.log"
Jul 16 22:44:55.450639 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.450552    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-cca4a2-n97q7_8afa8c6d-ad4b-4405-b4ce-9773ad1e12ed/revoke-keys/0.log"
Jul 16 22:44:55.664150 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.664120    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-601103b4-m2z2n_f5f80ade-9945-468b-8163-f480458ad49b/revoke-keys/0.log"
Jul 16 22:44:55.771030 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.770921    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-7ac0bc6b-gpk2p_0df82361-1574-45cf-bbf9-3805527c5d4b/revoke-keys/0.log"
Jul 16 22:44:55.884076 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.884047    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-1c9596-vkbcw_72451d71-b76e-4bd9-bad6-50e26726d0b9/revoke-keys/0.log"
Jul 16 22:44:55.990617 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:55.990584    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-7d2f73-2p2hf_f0bf07e1-f639-408a-bc8c-faba0eadb88d/revoke-keys/0.log"
Jul 16 22:44:57.948420 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:57.948384    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-szbm8_c598ac2d-2702-4336-bb1b-ce0fb66ed5a5/manager/0.log"
Jul 16 22:44:58.698436 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:58.698400    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-xfvv7_e1494e72-6683-4c57-a548-70f175887e9f/discovery/0.log"
Jul 16 22:44:59.981604 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:44:59.981576    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-76krz_c16bf3b4-7e21-44d6-bc4a-7f89f3acccab/main/0.log"
Jul 16 22:45:00.249490 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:00.249406    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxrp24_48a96291-b301-4202-b95f-d3d9bf38a5ed/main/0.log"
Jul 16 22:45:00.401640 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:00.401610    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tkmjx_8dc65c13-bf71-4849-875b-45293b86b4d3/main/0.log"
Jul 16 22:45:08.525634 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:08.525604    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c4l4q_186438be-cc8d-4091-897c-f83f9acd1c5c/global-pull-secret-syncer/0.log"
Jul 16 22:45:08.787157 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:08.787058    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pzkkb_2144dcd2-0967-4928-94ed-67201c9f687c/konnectivity-agent/0.log"
Jul 16 22:45:08.932841 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:08.932811    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-124.ec2.internal_1e21a0b953e6cd2a2d232f9eed463198/haproxy/0.log"
Jul 16 22:45:14.727472 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:14.727378    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-szbm8_c598ac2d-2702-4336-bb1b-ce0fb66ed5a5/manager/0.log"
Jul 16 22:45:17.300497 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.300463    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxhgh_90df56fe-e157-48f7-8303-9e5780fa8ce8/node-exporter/0.log"
Jul 16 22:45:17.329225 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.329196    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxhgh_90df56fe-e157-48f7-8303-9e5780fa8ce8/kube-rbac-proxy/0.log"
Jul 16 22:45:17.356716 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.356688    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxhgh_90df56fe-e157-48f7-8303-9e5780fa8ce8/init-textfile/0.log"
Jul 16 22:45:17.615474 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.615445    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac00706d-66c9-4892-a0a5-b7018dd478b1/prometheus/0.log"
Jul 16 22:45:17.641900 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.641873    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac00706d-66c9-4892-a0a5-b7018dd478b1/config-reloader/0.log"
Jul 16 22:45:17.676112 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.676050    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac00706d-66c9-4892-a0a5-b7018dd478b1/thanos-sidecar/0.log"
Jul 16 22:45:17.701137 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.701098    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac00706d-66c9-4892-a0a5-b7018dd478b1/kube-rbac-proxy-web/0.log"
Jul 16 22:45:17.725179 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.725152    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac00706d-66c9-4892-a0a5-b7018dd478b1/kube-rbac-proxy/0.log"
Jul 16 22:45:17.748701 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.748667    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac00706d-66c9-4892-a0a5-b7018dd478b1/kube-rbac-proxy-thanos/0.log"
Jul 16 22:45:17.774076 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:17.774052    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac00706d-66c9-4892-a0a5-b7018dd478b1/init-config-reloader/0.log"
Jul 16 22:45:18.313460 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313427    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"]
Jul 16 22:45:18.313848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313774    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerName="revoke-keys"
Jul 16 22:45:18.313848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313787    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerName="revoke-keys"
Jul 16 22:45:18.313848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313797    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:45:18.313848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313802    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f93b9d-1d90-45e9-ab85-40ea045bfdd4" containerName="revoke-keys"
Jul 16 22:45:18.313848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313821    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerName="revoke-keys"
Jul 16 22:45:18.313848 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313827    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerName="revoke-keys"
Jul 16 22:45:18.314081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313875    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerName="revoke-keys"
Jul 16 22:45:18.314081 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.313886    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac791d1-ea04-4a2e-8541-7fd541e0d9ab" containerName="revoke-keys"
Jul 16 22:45:18.316780 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.316765    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.321901 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.321876    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xsq47\"/\"kube-root-ca.crt\""
Jul 16 22:45:18.322977 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.322627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xsq47\"/\"default-dockercfg-8flnx\""
Jul 16 22:45:18.322977 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.322656    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xsq47\"/\"openshift-service-ca.crt\""
Jul 16 22:45:18.335290 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.335263    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"]
Jul 16 22:45:18.469063 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.469026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-sys\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.469259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.469080    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-podres\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.469259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.469184    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-lib-modules\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.469259 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.469226    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwww\" (UniqueName: \"kubernetes.io/projected/4fbd3c0a-f384-4321-ba4c-0471570171bf-kube-api-access-rpwww\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.469396 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.469276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-proc\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-sys\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570459 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-podres\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570720 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-lib-modules\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570720 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-sys\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570720 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwww\" (UniqueName: \"kubernetes.io/projected/4fbd3c0a-f384-4321-ba4c-0471570171bf-kube-api-access-rpwww\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570720 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570563    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-proc\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570720 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-lib-modules\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570720 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570600    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-podres\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.570720 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.570657    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbd3c0a-f384-4321-ba4c-0471570171bf-proc\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.580500 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.580469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwww\" (UniqueName: \"kubernetes.io/projected/4fbd3c0a-f384-4321-ba4c-0471570171bf-kube-api-access-rpwww\") pod \"perf-node-gather-daemonset-t4g6c\" (UID: \"4fbd3c0a-f384-4321-ba4c-0471570171bf\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.626571 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.626536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:18.759325 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:18.759300    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"]
Jul 16 22:45:18.761052 ip-10-0-143-124 kubenswrapper[2555]: W0716 22:45:18.761022    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4fbd3c0a_f384_4321_ba4c_0471570171bf.slice/crio-6a65d9719cc55a167988a4e38b24d0d972e9b8349d33040f50c5d89f74cfaa87 WatchSource:0}: Error finding container 6a65d9719cc55a167988a4e38b24d0d972e9b8349d33040f50c5d89f74cfaa87: Status 404 returned error can't find the container with id 6a65d9719cc55a167988a4e38b24d0d972e9b8349d33040f50c5d89f74cfaa87
Jul 16 22:45:19.022737 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:19.022701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c" event={"ID":"4fbd3c0a-f384-4321-ba4c-0471570171bf","Type":"ContainerStarted","Data":"e0d8b44157508efeab7c5b561994c1c9b9bc7ae6f5ce5bd75b5c0c28de816443"}
Jul 16 22:45:19.022941 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:19.022763    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c" event={"ID":"4fbd3c0a-f384-4321-ba4c-0471570171bf","Type":"ContainerStarted","Data":"6a65d9719cc55a167988a4e38b24d0d972e9b8349d33040f50c5d89f74cfaa87"}
Jul 16 22:45:19.022941 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:19.022868    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:19.041265 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:19.041210    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c" podStartSLOduration=1.041194881 podStartE2EDuration="1.041194881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:19.040016528 +0000 UTC m=+3475.033180677" watchObservedRunningTime="2026-07-16 22:45:19.041194881 +0000 UTC m=+3475.034359033"
Jul 16 22:45:19.305125 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:19.305047    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-hlgk5_49044b9b-aac9-4ff6-ac8b-b10a40b841f6/networking-console-plugin/0.log"
Jul 16 22:45:22.376383 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:22.376352    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xshcr_994e4bb0-4168-40b6-9a28-f73dcda6c9a0/dns/0.log"
Jul 16 22:45:22.402602 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:22.402577    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xshcr_994e4bb0-4168-40b6-9a28-f73dcda6c9a0/kube-rbac-proxy/0.log"
Jul 16 22:45:22.459591 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:22.459560    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rxtx7_86f80ab0-9e2d-4881-b1c1-71fd4a113f8e/dns-node-resolver/0.log"
Jul 16 22:45:23.149229 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:23.149203    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8rmsf_73baff7b-e485-492f-97da-09da89b0cc69/node-ca/0.log"
Jul 16 22:45:24.354463 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:24.354420    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-xfvv7_e1494e72-6683-4c57-a548-70f175887e9f/discovery/0.log"
Jul 16 22:45:25.038169 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:25.038141    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-t4g6c"
Jul 16 22:45:25.437260 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:25.437231    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v76p8_2d878d54-d241-49e1-ac1c-817462ac0e17/serve-healthcheck-canary/0.log"
Jul 16 22:45:26.156999 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:26.156969    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89z2g_cbe6c5ec-9f02-4dfe-b33f-c47f12527380/kube-rbac-proxy/0.log"
Jul 16 22:45:26.192860 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:26.192825    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89z2g_cbe6c5ec-9f02-4dfe-b33f-c47f12527380/exporter/0.log"
Jul 16 22:45:26.229950 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:26.229919    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89z2g_cbe6c5ec-9f02-4dfe-b33f-c47f12527380/extractor/0.log"
Jul 16 22:45:29.158011 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:29.157952    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-xfvv7_e1494e72-6683-4c57-a548-70f175887e9f/discovery/0.log"
Jul 16 22:45:29.542561 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:29.542511    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-77cfc_3c33481c-6a26-4613-98f7-df5125cf3b05/manager/0.log"
Jul 16 22:45:29.560804 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:29.560773    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-77cfc_3c33481c-6a26-4613-98f7-df5125cf3b05/copy-manifests/0.log"
Jul 16 22:45:30.821480 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:30.821444    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-xfvv7_e1494e72-6683-4c57-a548-70f175887e9f/discovery/0.log"
Jul 16 22:45:30.986747 ip-10-0-143-124 kubenswrapper[2555]: I0716 22:45:30.986711    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bfbbdbb89-tdkgr_fd74f09b-c195-4619-b8dd-f2a00231df4b/manager/0.log"