Jul 18 00:38:41.104521 ip-10-0-141-161 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 18 00:38:41.104533 ip-10-0-141-161 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 18 00:38:41.104542 ip-10-0-141-161 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 18 00:38:41.104989 ip-10-0-141-161 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 18 00:38:51.255625 ip-10-0-141-161 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 18 00:38:51.255641 ip-10-0-141-161 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1bacda9a917b45ab8c2d0f969d500ff5 --
Jul 18 00:40:43.774929 ip-10-0-141-161 systemd[1]: Starting Kubernetes Kubelet...
Jul 18 00:40:44.104462 ip-10-0-141-161 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 00:40:44.104462 ip-10-0-141-161 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 18 00:40:44.104462 ip-10-0-141-161 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 00:40:44.104462 ip-10-0-141-161 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 18 00:40:44.104462 ip-10-0-141-161 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 00:40:44.105979 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.105890    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 18 00:40:44.109669 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109649    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:44.109669 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109664    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:44.109669 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109668    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:44.109669 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109671    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:44.109669 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109675    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:44.109669 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109678    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109681    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109684    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109689    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109693    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109696    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109699    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109702    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109704    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109707    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109710    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109712    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109715    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109730    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109734    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109739    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109742    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109745    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109748    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:44.109885 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109752    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109755    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109758    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109761    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109763    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109766    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109769    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109771    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109775    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109778    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109782    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109785    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109789    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109793    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109796    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109799    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109801    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109804    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109807    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:44.110358 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109809    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109812    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109815    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109818    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109821    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109823    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109827    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109829    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109832    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109835    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109837    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109839    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109842    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109844    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109847    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109850    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109852    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109855    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109857    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109860    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:44.110834 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109862    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109865    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109867    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109869    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109872    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109876    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109878    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109881    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109884    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109887    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109889    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109892    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109895    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109897    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109900    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109903    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109905    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109908    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109910    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109913    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:44.111310 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109915    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109918    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109921    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109924    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109926    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109929    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109931    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109934    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109936    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109939    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109941    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109944    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109947    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109949    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109952    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.109954    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110707    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110716    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110719    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110722    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:44.111804 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110726    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110729    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110731    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110734    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110736    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110739    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110741    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110744    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110747    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110750    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110752    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110755    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110821    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110825    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110829    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110832    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110835    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110838    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110841    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110844    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:44.112288 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110847    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110850    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110852    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110855    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110858    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110860    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110863    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110866    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110869    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110872    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110874    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110878    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110883    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110886    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110889    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110891    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110894    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110897    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110899    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:44.112819 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110902    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110904    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110908    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110912    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110914    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110917    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110919    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110922    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110926    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110928    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110931    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110933    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110936    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110938    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110941    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110943    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110945    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110948    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110950    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110953    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:44.113276 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110955    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110958    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110960    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110963    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110966    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110969    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110971    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110974    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110977    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110979    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110982    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110984    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110987    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110989    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110995    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.110998    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111001    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111004    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111006    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:44.113765 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111009    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111011    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111014    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111017    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111019    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111022    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111024    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111027    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111029    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111032    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111034    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111037    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111040    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111042    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111045    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111047    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111050    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111458    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111465    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111468    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:44.114214 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111472    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111475    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111477    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111480    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111483    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111486    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111488    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111491    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111494    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111496    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111499    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111501    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111504    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111506    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111510    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111513    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111516    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111519    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111521    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111524    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:44.114717 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111526    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111529    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111531    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111534    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111536    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111539    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111541    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111544    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111546    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111549    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111552    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111555    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111558    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111561    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111563    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111566    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111569    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111571    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111574    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111577    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:44.115203 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111579    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111582    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111584    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111587    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111589    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111592    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111594    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111597    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111600    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111603    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111605    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111609    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111628    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111632    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111635    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111638    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111641    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111644    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111646    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:44.115719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111649    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111652    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111655    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111658    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111663    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111667    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111670    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111673    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111676    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111678    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111681    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111684    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111686    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111689    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111691    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111694    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111697    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111699    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111702    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:44.116179 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111704    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111707    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111710    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111712    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111715    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111717    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111720    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111722    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111725    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111728    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111731    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111734    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111737    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111740    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111742    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111744    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111747    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.111750    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111822    2554 flags.go:64] FLAG: --address="0.0.0.0"
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111829    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 18 00:40:44.116647 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111835    2554 flags.go:64] FLAG: --anonymous-auth="true"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111840    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111845    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111849    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111853    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111860    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111863    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111866    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111870    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111873    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111876    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111880    2554 flags.go:64] FLAG: --cgroup-root=""
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111883    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111886    2554 flags.go:64] FLAG: --client-ca-file=""
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111889    2554 flags.go:64] FLAG: --cloud-provider="external"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111892    2554 flags.go:64] FLAG: --cluster-dns="[]"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111897    2554 flags.go:64] FLAG: --cluster-domain=""
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111900    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111904    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111907    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111911    2554 flags.go:64] FLAG: --container-log-max-files="5"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111915    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111918    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 18 00:40:44.117119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111921    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111924    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111928    2554 flags.go:64] FLAG: --contention-profiling="false"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111931    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111934    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111938    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111941    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111946    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111949    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111952    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111955    2554 flags.go:64] FLAG: --enable-load-reader="false"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111958    2554 flags.go:64] FLAG: --enable-server="true"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111961    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111965    2554 flags.go:64] FLAG: --event-burst="100"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111968    2554 flags.go:64] FLAG: --event-qps="50"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111971    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111974    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111977    2554 flags.go:64] FLAG: --eviction-hard=""
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111981    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111984    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111987    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111990    2554 flags.go:64] FLAG: --eviction-soft=""
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111993    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111995    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.111998    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 18 00:40:44.117660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112002    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112005    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112008    2554 flags.go:64] FLAG: --fail-swap-on="true"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112011    2554 flags.go:64] FLAG: --feature-gates=""
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112015    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112018    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112021    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112024    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112027    2554 flags.go:64] FLAG: --healthz-port="10248"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112036    2554 flags.go:64] FLAG: --help="false"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112039    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112042    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112045    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112048    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112051    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112054    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112057    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112060    2554 flags.go:64] FLAG: --image-service-endpoint=""
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112063    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112066    2554 flags.go:64] FLAG: --kube-api-burst="100"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112069    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112076    2554 flags.go:64] FLAG: --kube-api-qps="50"
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112079    2554 flags.go:64] FLAG: --kube-reserved=""
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112082    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 18 00:40:44.118318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112085    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112088    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112091    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112094    2554 flags.go:64] FLAG: --lock-file=""
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112097    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112100    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112102    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112108    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112111    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112114    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112120    2554 flags.go:64] FLAG: --logging-format="text"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112123    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112127    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112130    2554 flags.go:64] FLAG: --manifest-url=""
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112133    2554 flags.go:64] FLAG: --manifest-url-header=""
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112137    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112140    2554 flags.go:64] FLAG: --max-open-files="1000000"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112146    2554 flags.go:64] FLAG: --max-pods="110"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112149    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112152    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112155    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112158    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112161    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112164    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112167    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 18 00:40:44.118916 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112174    2554 flags.go:64] FLAG: --node-status-max-images="50"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112177    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112180    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112183    2554 flags.go:64] FLAG: --pod-cidr=""
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112186    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112192    2554 flags.go:64] FLAG: --pod-manifest-path=""
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112194    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112198    2554 flags.go:64] FLAG: --pods-per-core="0"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112200    2554 flags.go:64] FLAG: --port="10250"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112203    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112206    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b08818d4b12635af"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112209    2554 flags.go:64] FLAG: --qos-reserved=""
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112213    2554 flags.go:64] FLAG: --read-only-port="10255"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112216    2554 flags.go:64] FLAG: --register-node="true"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112218    2554 flags.go:64] FLAG: --register-with-taints=""
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112222    2554 flags.go:64] FLAG: --registry-burst="10"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112225    2554 flags.go:64] FLAG: --registry-qps="5"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112228    2554 flags.go:64] FLAG: --reserved-cpus=""
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112233    2554 flags.go:64] FLAG: --reserved-memory=""
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112237    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112240    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112242    2554 flags.go:64] FLAG: --rotate-certificates="false"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112245    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112249    2554 flags.go:64] FLAG: --runonce="false"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112253    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 18 00:40:44.119737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112256    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112259    2554 flags.go:64] FLAG: --seccomp-default="false"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112262    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112264    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112267    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112270    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112273    2554 flags.go:64] FLAG: --storage-driver-password="root"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112276    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112279    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112282    2554 flags.go:64] FLAG: --storage-driver-user="root"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112285    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112288    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112291    2554 flags.go:64] FLAG: --system-cgroups=""
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112294    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112299    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112302    2554 flags.go:64] FLAG: --tls-cert-file=""
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112305    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112310    2554 flags.go:64] FLAG: --tls-min-version=""
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112313    2554 flags.go:64] FLAG: --tls-private-key-file=""
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112316    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112318    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112321    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112324    2554 flags.go:64] FLAG: --v="2"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112328    2554 flags.go:64] FLAG: --version="false"
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112332    2554 flags.go:64] FLAG: --vmodule=""
Jul 18 00:40:44.120655 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112338    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.112342    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112457    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112460    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112464    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112466    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112471    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112474    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112477    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112480    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112483    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112485    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112488    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112491    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112494    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112497    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112501    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112504    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112506    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112509    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:44.121248 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112512    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112514    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112517    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112520    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112522    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112525    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112527    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112530    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112534    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112538    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112541    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112544    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112547    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112551    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112553    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112556    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112558    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112561    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112565    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:44.121848 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112567    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112569    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112572    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112574    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112577    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112580    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112582    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112585    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112588    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112590    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112593    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112595    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112598    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112600    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112603    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112605    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112608    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112611    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112627    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112630    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:44.122343 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112633    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112636    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112638    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112641    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112644    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112647    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112650    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112653    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112656    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112659    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112661    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112665    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112668    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112670    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112673    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112676    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112678    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112681    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112683    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112686    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:44.122843 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112688    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112691    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112693    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112696    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112699    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112702    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112706    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112709    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112712    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112715    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112719    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112722    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112725    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112728    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112731    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112734    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112737    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112741    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112744    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:44.123332 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112747    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112750    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.112753    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.113259    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.121024    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.121045    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.121817    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.121825    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121884    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121889    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121893    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121896    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121899    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121902    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121904    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:44.123811 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121907    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121910    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121912    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121915    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121917    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121920    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121922    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121925    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121928    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121933    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121936    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121940    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121944    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121947    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121950    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121953    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121956    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121959    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121961    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:44.124177 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121965    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121969    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121972    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121976    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121979    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121982    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121985    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121987    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121990    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121993    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121996    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.121998    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122000    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122003    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122006    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122008    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122011    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122014    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122016    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:44.124711 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122019    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122022    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122025    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122027    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122029    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122032    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122035    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122037    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122040    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122042    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122045    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122048    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122051    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122055    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122058    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122061    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122064    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122066    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122069    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122072    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:44.125218 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122074    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122076    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122079    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122081    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122084    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122086    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122089    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122091    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122094    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122097    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122099    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122102    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122104    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122106    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122110    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122112    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122115    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122118    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122120    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122122    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:44.125705 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122125    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122128    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122130    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122133    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122135    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122138    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122140    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122143    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122147    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122149    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122152    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122154    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122157    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122160    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.122165    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 00:40:44.126181 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122290    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122295    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122297    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122300    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122303    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122305    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122308    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122311    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122314    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122317    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122319    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122322    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122326    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122328    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122331    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122333    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122336    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122338    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122341    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:44.126551 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122344    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122347    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122350    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122352    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122356    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122360    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122364    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122367    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122370    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122372    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122375    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122377    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122380    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122382    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122385    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122387    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122389    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122393    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122396    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:44.127022 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122399    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122403    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122405    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122408    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122410    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122413    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122416    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122419    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122422    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122424    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122427    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122429    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122432    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122434    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122437    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122440    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122442    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122445    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122447    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122450    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:44.127474 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122453    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122456    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122458    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122461    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122463    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122466    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122468    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122471    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122473    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122476    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122478    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122481    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122483    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122486    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122488    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122491    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122493    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122496    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122498    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122501    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:44.127967 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122506    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122509    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122512    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122514    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122516    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122519    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122522    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122524    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122526    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122529    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122531    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122534    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122537    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122539    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122542    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122544    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122547    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122549    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122553    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122556    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:44.128466 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:44.122559    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:44.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.122564    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 00:40:44.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.122689    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 18 00:40:44.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.124708    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 18 00:40:44.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.125832    2554 server.go:1019] "Starting client certificate rotation"
Jul 18 00:40:44.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.125934    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 18 00:40:44.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.125963    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 18 00:40:44.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.128085    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 18 00:40:44.129280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.129264    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 18 00:40:44.142092 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.142069    2554 log.go:25] "Validated CRI v1 runtime API"
Jul 18 00:40:44.144057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.144038    2554 log.go:25] "Validated CRI v1 image API"
Jul 18 00:40:44.144817 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.144804    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 18 00:40:44.147337 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.147307    2554 fs.go:135] Filesystem UUIDs: map[1d4b189a-1b58-41f9-984f-4b1b6ad6d843:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e4cc17a4-0216-4320-811a-027ea1e8f93a:/dev/nvme0n1p4]
Jul 18 00:40:44.147389 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.147338    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 18 00:40:44.153726 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.153018    2554 manager.go:216] Machine: {Timestamp:2026-07-18 00:40:44.151059423 +0000 UTC m=+0.276994392 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101511 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dbce75fd4d8926b13b5c49c8be944 SystemUUID:ec2dbce7-5fd4-d892-6b13-b5c49c8be944 BootID:1bacda9a-917b-45ab-8c2d-0f969d500ff5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:9e:eb:d4:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:9e:eb:d4:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:ac:6a:94:4e:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 18 00:40:44.153726 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.153233    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 18 00:40:44.153726 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.153374    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 18 00:40:44.154733 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.154440    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 18 00:40:44.154791 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.154731    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 18 00:40:44.154870 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.154740    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 18 00:40:44.155592 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.155581    2554 topology_manager.go:138] "Creating topology manager with none policy"
Jul 18 00:40:44.155643 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.155595    2554 container_manager_linux.go:309] "Creating device plugin manager"
Jul 18 00:40:44.155643 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.155603    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 18 00:40:44.156221 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.156211    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 18 00:40:44.157344 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.157334    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 18 00:40:44.161059 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161047    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 18 00:40:44.161173 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161164    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 18 00:40:44.161205 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161194    2554 kubelet.go:486] "Attempting to sync node with API server"
Jul 18 00:40:44.161205 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161204    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 18 00:40:44.161263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161219    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 18 00:40:44.161263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161228    2554 kubelet.go:398] "Adding apiserver pod source"
Jul 18 00:40:44.161263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161236    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 18 00:40:44.161604 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161591    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 18 00:40:44.161604 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.161611    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 18 00:40:44.163431 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.163410    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 18 00:40:44.165132 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.165120    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 18 00:40:44.165192 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.165150    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 18 00:40:44.166178 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166161    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 18 00:40:44.166178 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166180    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166188    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166196    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166207    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166215    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166226    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166233    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166243    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166251    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166264    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 18 00:40:44.166418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.166272    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 18 00:40:44.173184 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.173159    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 18 00:40:44.173184 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.173159    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 18 00:40:44.176251 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.176234    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 18 00:40:44.176251 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.176255    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 18 00:40:44.179378 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.179362    2554 server.go:1268] "Started kubelet"
Jul 18 00:40:44.179502 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.179458    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 18 00:40:44.179724 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.179440    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 18 00:40:44.179724 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.179592    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 18 00:40:44.179996 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.179966    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ttkgh"
Jul 18 00:40:44.180274 ip-10-0-141-161 systemd[1]: Started Kubernetes Kubelet.
Jul 18 00:40:44.180749 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.180731    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 18 00:40:44.181088 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.181067    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 18 00:40:44.181369 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.181356    2554 server.go:310] "Adding debug handlers to kubelet server"
Jul 18 00:40:44.184381 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.184347    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 18 00:40:44.185494 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.185476    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ttkgh"
Jul 18 00:40:44.187133 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.187117    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 18 00:40:44.187266 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.186530    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-161.ec2.internal.18c33b736584e8c3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-161.ec2.internal,UID:ip-10-0-141-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-161.ec2.internal,},FirstTimestamp:2026-07-18 00:40:44.179335363 +0000 UTC m=+0.305270333,LastTimestamp:2026-07-18 00:40:44.179335363 +0000 UTC m=+0.305270333,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-161.ec2.internal,}"
Jul 18 00:40:44.187561 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.187532    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 18 00:40:44.187682 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.187659    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.188342 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188103    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 18 00:40:44.188342 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188107    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 18 00:40:44.188342 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188129    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 18 00:40:44.188342 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188229    2554 reconstruct.go:97] "Volume reconstruction finished"
Jul 18 00:40:44.188342 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188242    2554 reconciler.go:29] "Reconciler: start to sync state"
Jul 18 00:40:44.188559 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188545    2554 factory.go:55] Registering systemd factory
Jul 18 00:40:44.188644 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188563    2554 factory.go:223] Registration of the systemd container factory successfully
Jul 18 00:40:44.188889 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188870    2554 factory.go:153] Registering CRI-O factory
Jul 18 00:40:44.188889 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188890    2554 factory.go:223] Registration of the crio container factory successfully
Jul 18 00:40:44.189018 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188937    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 18 00:40:44.189018 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188955    2554 factory.go:103] Registering Raw factory
Jul 18 00:40:44.189018 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.188970    2554 manager.go:1196] Started watching for new ooms in manager
Jul 18 00:40:44.189418 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.189406    2554 manager.go:349] Starting recovery of all containers
Jul 18 00:40:44.190541 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.190519    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 18 00:40:44.190691 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.190667    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 18 00:40:44.202361 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.202334    2554 manager.go:354] Recovery completed
Jul 18 00:40:44.207392 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.207379    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:44.209186 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.209164    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:44.209268 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.209206    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:44.209268 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.209234    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:44.209866 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.209849    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 18 00:40:44.209866 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.209864    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 18 00:40:44.210006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.209884    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 18 00:40:44.210006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.209915    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:44.210692 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.210677    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:44.210767 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.210707    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:44.210767 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.210721    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:44.211995 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.211981    2554 policy_none.go:49] "None policy: Start"
Jul 18 00:40:44.212060 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.211999    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 18 00:40:44.212060 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.212012    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 18 00:40:44.213940 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.213927    2554 policy_none.go:47] "Start"
Jul 18 00:40:44.250319 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.250145    2554 manager.go:337] "Starting Device Plugin manager"
Jul 18 00:40:44.250433 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.250346    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 18 00:40:44.250433 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.250362    2554 server.go:87] "Starting device plugin registration server"
Jul 18 00:40:44.250657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.250642    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 18 00:40:44.250716 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.250655    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 18 00:40:44.250768 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.250746    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 18 00:40:44.251312 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.250844    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 18 00:40:44.251312 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.250859    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 18 00:40:44.251467 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.251381    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 18 00:40:44.251467 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.251425    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.328445 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.328396    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 18 00:40:44.330596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.329611    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 18 00:40:44.330596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.329687    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 18 00:40:44.330596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.329716    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 18 00:40:44.330596 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.329770    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 18 00:40:44.331832 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.331811    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:44.351567 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.351543    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:44.355843 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.355802    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:44.355843 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.355831    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:44.355947 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.355844    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:44.355947 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.355870    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.363312 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.363297    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.363374 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.363330    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-161.ec2.internal\": node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.380297 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.380276    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.430841 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.430810    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal"]
Jul 18 00:40:44.430924 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.430881    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:44.431912 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.431894    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:44.431996 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.431924    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:44.431996 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.431934    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:44.434212 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.434200    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:44.434346 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.434330    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.434405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.434366    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:44.435003 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.434987    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:44.435100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.435019    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:44.435100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.434991    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:44.435100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.435034    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:44.435100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.435060    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:44.435100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.435075    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:44.437265 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.437249    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.437345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.437277    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:44.438092 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.438076    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:44.438174 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.438104    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:44.438174 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.438117    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:44.455201 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.455179    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-161.ec2.internal\" not found" node="ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.461251 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.461235    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-161.ec2.internal\" not found" node="ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.480934 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.480909    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.489298 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.489268    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8cf2fde091123b7b6cb170fddd088984-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal\" (UID: \"8cf2fde091123b7b6cb170fddd088984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.489298 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.489296    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cf2fde091123b7b6cb170fddd088984-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal\" (UID: \"8cf2fde091123b7b6cb170fddd088984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.489445 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.489316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a0ab874ed9de1072b24764a5ad02232-config\") pod \"kube-apiserver-proxy-ip-10-0-141-161.ec2.internal\" (UID: \"6a0ab874ed9de1072b24764a5ad02232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.581088 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.581057    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.589999 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.589974    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8cf2fde091123b7b6cb170fddd088984-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal\" (UID: \"8cf2fde091123b7b6cb170fddd088984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.590079 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.590007    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cf2fde091123b7b6cb170fddd088984-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal\" (UID: \"8cf2fde091123b7b6cb170fddd088984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.590079 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.590035    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a0ab874ed9de1072b24764a5ad02232-config\") pod \"kube-apiserver-proxy-ip-10-0-141-161.ec2.internal\" (UID: \"6a0ab874ed9de1072b24764a5ad02232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.590148 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.590075    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8cf2fde091123b7b6cb170fddd088984-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal\" (UID: \"8cf2fde091123b7b6cb170fddd088984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.590148 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.590085    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cf2fde091123b7b6cb170fddd088984-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal\" (UID: \"8cf2fde091123b7b6cb170fddd088984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.590148 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.590084    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a0ab874ed9de1072b24764a5ad02232-config\") pod \"kube-apiserver-proxy-ip-10-0-141-161.ec2.internal\" (UID: \"6a0ab874ed9de1072b24764a5ad02232\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.682412 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.682322    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.762590 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.762555    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.763741 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:44.763722    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:44.782733 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.782709    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.889400 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.889345    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:44.990146 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:44.990050    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:45.088283 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.088251    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:45.091095 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:45.091065    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:45.126960 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.126931    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 18 00:40:45.127463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.127083    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 18 00:40:45.127463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.127136    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 18 00:40:45.192080 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:45.192048    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:45.192080 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.192064    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 00:35:44 +0000 UTC" deadline="2028-03-04 21:53:57.334574444 +0000 UTC"
Jul 18 00:40:45.192267 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.192090    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14301h13m12.142487339s"
Jul 18 00:40:45.292956 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:45.292910    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-161.ec2.internal\" not found"
Jul 18 00:40:45.312987 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.312959    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:45.379398 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:45.379353    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0ab874ed9de1072b24764a5ad02232.slice/crio-3ad5be47c162f4464083bbcbfa728b2b079199b8cba9f3ac706e78df498237e7 WatchSource:0}: Error finding container 3ad5be47c162f4464083bbcbfa728b2b079199b8cba9f3ac706e78df498237e7: Status 404 returned error can't find the container with id 3ad5be47c162f4464083bbcbfa728b2b079199b8cba9f3ac706e78df498237e7
Jul 18 00:40:45.387824 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.387793    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:45.387928 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.387884    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 00:40:45.394478 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:45.394452    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf2fde091123b7b6cb170fddd088984.slice/crio-edc574399a04e2e2fc6e40af123e000c121e5b0c23dc2ca1486a0df34466c43e WatchSource:0}: Error finding container edc574399a04e2e2fc6e40af123e000c121e5b0c23dc2ca1486a0df34466c43e: Status 404 returned error can't find the container with id edc574399a04e2e2fc6e40af123e000c121e5b0c23dc2ca1486a0df34466c43e
Jul 18 00:40:45.400661 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.400643    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 18 00:40:45.401497 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.401481    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal"
Jul 18 00:40:45.410935 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.410912    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 18 00:40:45.430519 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:45.430496    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:46.109491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.109342    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:46.162643 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.162598    2554 apiserver.go:52] "Watching apiserver"
Jul 18 00:40:46.171068 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.171044    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 18 00:40:46.171523 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.171494    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal","openshift-image-registry/node-ca-tjbks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal","openshift-multus/multus-546f9","openshift-network-operator/iptables-alerter-p9wrt","kube-system/global-pull-secret-syncer-tc4gd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj","openshift-cluster-node-tuning-operator/tuned-jwzrf","openshift-dns/node-resolver-9jvs2","openshift-multus/multus-additional-cni-plugins-9k5md","openshift-multus/network-metrics-daemon-4p62b","openshift-network-diagnostics/network-check-target-qqqj9","openshift-ovn-kubernetes/ovnkube-node-dg4s4","kube-system/konnectivity-agent-t96fg"]
Jul 18 00:40:46.174213 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.174190    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.176811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.176435    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 18 00:40:46.176811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.176529    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 18 00:40:46.176811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.176689    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkjgn\""
Jul 18 00:40:46.177011 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.176863    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 18 00:40:46.177011 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.176902    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.179592 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.179057    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.180253 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.180164    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 18 00:40:46.180253 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.180223    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 18 00:40:46.181743 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.180170    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8466p\""
Jul 18 00:40:46.184450 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.181892    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 18 00:40:46.184450 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.181992    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 18 00:40:46.184450 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.182037    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 18 00:40:46.184450 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.183368    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 18 00:40:46.184450 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.184087    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 18 00:40:46.184742 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.184463    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7vh6m\""
Jul 18 00:40:46.184742 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.184672    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 18 00:40:46.186340 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.185569    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.188808 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.188754    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 18 00:40:46.189206 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.189039    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xmvdr\""
Jul 18 00:40:46.189324 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.189308    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 18 00:40:46.189459 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.189444    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.189589 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.189573    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 18 00:40:46.189888 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.189865    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 18 00:40:46.191497 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.191478    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 18 00:40:46.191862 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.191775    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.193020 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.192992    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:40:46.193819 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.193798    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 18 00:40:46.194016 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.193988    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 18 00:40:46.194121 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.194068    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6sfkd\""
Jul 18 00:40:46.195644 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.195608    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.196403 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.196385    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 18 00:40:46.197932 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.197797    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 18 00:40:46.198512 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.198490    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.198637 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.198550    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 18 00:40:46.198637 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.198591    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6ll72\""
Jul 18 00:40:46.198964 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.198938    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-log-socket\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.199034 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.198978    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovnkube-config\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.199034 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199004    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-env-overrides\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.199034 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199026    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-system-cni-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199181 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199048    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.199181 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199070    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-device-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.199181 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199093    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-var-lib-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.199181 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199138    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-socket-dir-parent\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199167    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-k8s-cni-cncf-io\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199239    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-hostroot\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199262    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-multus-certs\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199285    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmf27\" (UniqueName: \"kubernetes.io/projected/761c310d-2fe7-4644-aec4-2182615102af-kube-api-access-kmf27\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199310    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a089e7e1-1597-4899-83b2-357a35bb40cb-dbus\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199333    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199356    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-socket-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.199380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-sys-fs\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199400    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-etc-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199433    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovnkube-script-lib\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199495    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-cnibin\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199519    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97a32a19-9383-4228-9d56-a207a9ebb4cb-host-slash\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a089e7e1-1597-4899-83b2-357a35bb40cb-kubelet-config\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199592    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0621ebeb-5101-4564-9c03-a208a0a105f7-konnectivity-ca\") pod \"konnectivity-agent-t96fg\" (UID: \"0621ebeb-5101-4564-9c03-a208a0a105f7\") " pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-cni-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199660    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-netns\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199726    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-cni-multus\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199767    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-kubelet\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.199811 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199801    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/761c310d-2fe7-4644-aec4-2182615102af-multus-daemon-config\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199827    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-kubelet\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199856    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0621ebeb-5101-4564-9c03-a208a0a105f7-agent-certs\") pod \"konnectivity-agent-t96fg\" (UID: \"0621ebeb-5101-4564-9c03-a208a0a105f7\") " pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199919    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-os-release\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.199949    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-conf-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200024    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jrr\" (UniqueName: \"kubernetes.io/projected/97a32a19-9383-4228-9d56-a207a9ebb4cb-kube-api-access-p4jrr\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200070    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200105    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-cni-bin\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200137    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200161    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzcs\" (UniqueName: \"kubernetes.io/projected/05fbcf08-8b1c-476d-9814-a803fe24cfcd-kube-api-access-sjzcs\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200187    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-etc-kubernetes\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200214    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-systemd\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200241    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/761c310d-2fe7-4644-aec4-2182615102af-cni-binary-copy\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200266    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-cni-bin\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200291    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-registration-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.200330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200314    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7cs\" (UniqueName: \"kubernetes.io/projected/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-kube-api-access-tv7cs\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200367    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-systemd-units\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200390    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-slash\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-node-log\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200445    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200487    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-cni-netd\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200518    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97a32a19-9383-4228-9d56-a207a9ebb4cb-iptables-alerter-script\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200547    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-run-netns\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200569    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200594    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-ovn\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200631    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200834    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xmj6x\""
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200870    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 18 00:40:46.201006 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.200947    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.201974 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.201887    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 18 00:40:46.203507 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.202945    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 18 00:40:46.203507 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.203258    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 18 00:40:46.203507 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.203349    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 18 00:40:46.203507 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.203466    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7t2bp\""
Jul 18 00:40:46.204063 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.204043    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.206577 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.206497    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8hpx\""
Jul 18 00:40:46.206577 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.206564    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 18 00:40:46.206891 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.206732    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 18 00:40:46.208013 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.207993    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:46.208101 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.208069    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:40:46.210412 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.210336    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:46.210412 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.210387    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:40:46.224234 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.224196    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x7v6z"
Jul 18 00:40:46.232534 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.232509    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x7v6z"
Jul 18 00:40:46.290257 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.290226    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 18 00:40:46.300870 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.300839    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysctl-d\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.301033 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.300878    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0621ebeb-5101-4564-9c03-a208a0a105f7-konnectivity-ca\") pod \"konnectivity-agent-t96fg\" (UID: \"0621ebeb-5101-4564-9c03-a208a0a105f7\") " pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.301033 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.300896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-cni-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301033 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.300918    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-netns\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301192 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301046    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-netns\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301192 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301135    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-os-release\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.301192 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301165    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-systemd\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.301330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301218    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-run\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.301330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301234    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-cni-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-var-lib-kubelet\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.301330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301275    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-host\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.301330 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301293    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-kubelet\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301329    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301371    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-os-release\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301395    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-conf-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301407    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-kubelet\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/482bb2e3-16af-4029-b9cb-4ce88e7cc092-host\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301426    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301457    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794706ac-dc6b-451e-9dfc-688409907cf9-tmp-dir\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301474    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-conf-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301492    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-os-release\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-system-cni-dir\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301528    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6rmf\" (UniqueName: \"kubernetes.io/projected/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-kube-api-access-v6rmf\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301564    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jrr\" (UniqueName: \"kubernetes.io/projected/97a32a19-9383-4228-9d56-a207a9ebb4cb-kube-api-access-p4jrr\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301569    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0621ebeb-5101-4564-9c03-a208a0a105f7-konnectivity-ca\") pod \"konnectivity-agent-t96fg\" (UID: \"0621ebeb-5101-4564-9c03-a208a0a105f7\") " pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.301648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301606    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301735    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/761c310d-2fe7-4644-aec4-2182615102af-cni-binary-copy\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301753    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301794    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dzd\" (UniqueName: \"kubernetes.io/projected/482bb2e3-16af-4029-b9cb-4ce88e7cc092-kube-api-access-c6dzd\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301838    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/794706ac-dc6b-451e-9dfc-688409907cf9-hosts-file\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301867    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-kube-api-access-dldn9\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301919    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4tr\" (UniqueName: \"kubernetes.io/projected/2bd69c75-0071-4835-a044-9accda5ecee4-kube-api-access-tw4tr\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301945    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-node-log\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.301973    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302010    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-cni-netd\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302031    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-node-log\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302035    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-system-cni-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302077    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-cni-netd\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-system-cni-dir\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302075    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysctl-conf\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.302281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302126    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-modprobe-d\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302150    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysconfig\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302174    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-tmp\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302200    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302251    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302264    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/761c310d-2fe7-4644-aec4-2182615102af-cni-binary-copy\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302277    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-var-lib-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302314    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-k8s-cni-cncf-io\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302337    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-hostroot\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302382    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-k8s-cni-cncf-io\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302416    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-hostroot\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302431    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-var-lib-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302496    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-sys-fs\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302520    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-etc-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302543    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovnkube-script-lib\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302562    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-sys-fs\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.303057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302570    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-cni-multus\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-etc-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.302637    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/482bb2e3-16af-4029-b9cb-4ce88e7cc092-serviceca\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.302707    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret podName:a089e7e1-1597-4899-83b2-357a35bb40cb nodeName:}" failed. No retries permitted until 2026-07-18 00:40:46.802686693 +0000 UTC m=+2.928621653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret") pod "global-pull-secret-syncer-tc4gd" (UID: "a089e7e1-1597-4899-83b2-357a35bb40cb") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-cni-multus\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302782    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97a32a19-9383-4228-9d56-a207a9ebb4cb-host-slash\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302804    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a089e7e1-1597-4899-83b2-357a35bb40cb-kubelet-config\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302825    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0621ebeb-5101-4564-9c03-a208a0a105f7-agent-certs\") pod \"konnectivity-agent-t96fg\" (UID: \"0621ebeb-5101-4564-9c03-a208a0a105f7\") " pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302849    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-cni-bin\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302877    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302900    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzcs\" (UniqueName: \"kubernetes.io/projected/05fbcf08-8b1c-476d-9814-a803fe24cfcd-kube-api-access-sjzcs\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302929    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a089e7e1-1597-4899-83b2-357a35bb40cb-kubelet-config\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302975    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-cni-bin\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.302934    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cni-binary-copy\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303011    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97a32a19-9383-4228-9d56-a207a9ebb4cb-host-slash\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303038    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-kubernetes\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.303934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-tuned\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303094    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-systemd\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303120    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-cni-bin\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovnkube-script-lib\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303145    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303173    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303176    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-systemd\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303205    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-etc-kubernetes\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303212    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-cni-bin\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303232    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-registration-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303237    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-etc-kubernetes\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303257    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7cs\" (UniqueName: \"kubernetes.io/projected/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-kube-api-access-tv7cs\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303274    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303280    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-registration-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303282    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-systemd-units\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303312    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-systemd-units\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-slash\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.304634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303352    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-socket-dir-parent\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303367    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmf27\" (UniqueName: \"kubernetes.io/projected/761c310d-2fe7-4644-aec4-2182615102af-kube-api-access-kmf27\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97a32a19-9383-4228-9d56-a207a9ebb4cb-iptables-alerter-script\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303432    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-multus-socket-dir-parent\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303482    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-slash\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303520    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-run-netns\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303553    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-ovn\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303608    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-log-socket\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-host-run-netns\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303650    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-openvswitch\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303655    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovnkube-config\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303684    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-run-ovn\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303683    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-device-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303695    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05fbcf08-8b1c-476d-9814-a803fe24cfcd-log-socket\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303713    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-multus-certs\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303729    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-device-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.305491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-env-overrides\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303769    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6h8\" (UniqueName: \"kubernetes.io/projected/794706ac-dc6b-451e-9dfc-688409907cf9-kube-api-access-5k6h8\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303797    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-sys\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303853    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-lib-modules\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303883    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a089e7e1-1597-4899-83b2-357a35bb40cb-dbus\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303908    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-socket-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303909    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-run-multus-certs\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303930    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97a32a19-9383-4228-9d56-a207a9ebb4cb-iptables-alerter-script\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303939    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-cnibin\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-kubelet\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.303990    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/761c310d-2fe7-4644-aec4-2182615102af-multus-daemon-config\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304014    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cnibin\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304016    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a089e7e1-1597-4899-83b2-357a35bb40cb-dbus\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304110    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-socket-dir\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304116    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovnkube-config\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.306280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304123    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05fbcf08-8b1c-476d-9814-a803fe24cfcd-env-overrides\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.307051 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304127    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-cnibin\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.307051 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304151    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/761c310d-2fe7-4644-aec4-2182615102af-host-var-lib-kubelet\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.307051 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.304503    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/761c310d-2fe7-4644-aec4-2182615102af-multus-daemon-config\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.307347 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.307325    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05fbcf08-8b1c-476d-9814-a803fe24cfcd-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.307411 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.307329    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0621ebeb-5101-4564-9c03-a208a0a105f7-agent-certs\") pod \"konnectivity-agent-t96fg\" (UID: \"0621ebeb-5101-4564-9c03-a208a0a105f7\") " pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.311462 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.311416    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jrr\" (UniqueName: \"kubernetes.io/projected/97a32a19-9383-4228-9d56-a207a9ebb4cb-kube-api-access-p4jrr\") pod \"iptables-alerter-p9wrt\" (UID: \"97a32a19-9383-4228-9d56-a207a9ebb4cb\") " pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.312042 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.311992    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7cs\" (UniqueName: \"kubernetes.io/projected/40e3ae46-70d9-46c9-b8a1-73bae5d6437b-kube-api-access-tv7cs\") pod \"aws-ebs-csi-driver-node-fggvj\" (UID: \"40e3ae46-70d9-46c9-b8a1-73bae5d6437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.312521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.312499    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmf27\" (UniqueName: \"kubernetes.io/projected/761c310d-2fe7-4644-aec4-2182615102af-kube-api-access-kmf27\") pod \"multus-546f9\" (UID: \"761c310d-2fe7-4644-aec4-2182615102af\") " pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.313353 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.313309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzcs\" (UniqueName: \"kubernetes.io/projected/05fbcf08-8b1c-476d-9814-a803fe24cfcd-kube-api-access-sjzcs\") pod \"ovnkube-node-dg4s4\" (UID: \"05fbcf08-8b1c-476d-9814-a803fe24cfcd\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.334779 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.334723    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal" event={"ID":"8cf2fde091123b7b6cb170fddd088984","Type":"ContainerStarted","Data":"edc574399a04e2e2fc6e40af123e000c121e5b0c23dc2ca1486a0df34466c43e"}
Jul 18 00:40:46.336131 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.336105    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal" event={"ID":"6a0ab874ed9de1072b24764a5ad02232","Type":"ContainerStarted","Data":"3ad5be47c162f4464083bbcbfa728b2b079199b8cba9f3ac706e78df498237e7"}
Jul 18 00:40:46.405219 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6h8\" (UniqueName: \"kubernetes.io/projected/794706ac-dc6b-451e-9dfc-688409907cf9-kube-api-access-5k6h8\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.405219 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.405219 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405192    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-sys\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405263    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-sys\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405296    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-lib-modules\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405314    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cnibin\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405347    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysctl-d\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-os-release\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405407    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-systemd\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405425    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-run\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-var-lib-kubelet\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405461    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-host\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405491    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/482bb2e3-16af-4029-b9cb-4ce88e7cc092-host\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405508    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794706ac-dc6b-451e-9dfc-688409907cf9-tmp-dir\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405529    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-system-cni-dir\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rmf\" (UniqueName: \"kubernetes.io/projected/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-kube-api-access-v6rmf\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dzd\" (UniqueName: \"kubernetes.io/projected/482bb2e3-16af-4029-b9cb-4ce88e7cc092-kube-api-access-c6dzd\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/794706ac-dc6b-451e-9dfc-688409907cf9-hosts-file\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405587    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysctl-d\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405605    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405662    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-kube-api-access-dldn9\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405682    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4tr\" (UniqueName: \"kubernetes.io/projected/2bd69c75-0071-4835-a044-9accda5ecee4-kube-api-access-tw4tr\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405702    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysctl-conf\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405717    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-modprobe-d\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405720    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-systemd\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405732    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysconfig\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405749    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-tmp\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.405814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405779    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/482bb2e3-16af-4029-b9cb-4ce88e7cc092-serviceca\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405801    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cni-binary-copy\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405799    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405815    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-kubernetes\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405801    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-var-lib-kubelet\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405845    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-kubernetes\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405844    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysctl-conf\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405892    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-os-release\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405918    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-modprobe-d\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405918    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-host\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cnibin\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.405992    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794706ac-dc6b-451e-9dfc-688409907cf9-tmp-dir\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406045    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-lib-modules\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406070    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/482bb2e3-16af-4029-b9cb-4ce88e7cc092-host\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406102    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-run\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406270    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-sysconfig\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406278    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/794706ac-dc6b-451e-9dfc-688409907cf9-hosts-file\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406324    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-tuned\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.406542 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406399    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-system-cni-dir\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406417    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406431    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/482bb2e3-16af-4029-b9cb-4ce88e7cc092-serviceca\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406450    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406498    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.406534    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cni-binary-copy\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.406597    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs podName:2bd69c75-0071-4835-a044-9accda5ecee4 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:46.906579894 +0000 UTC m=+3.032514851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs") pod "network-metrics-daemon-4p62b" (UID: "2bd69c75-0071-4835-a044-9accda5ecee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:46.407421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.406973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.409030 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.409007    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-tmp\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.410149 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.410129    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-etc-tuned\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.412580 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.412557    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:46.412691 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.412585    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:46.412691 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.412598    2554 projected.go:196] Error preparing data for projected volume kube-api-access-pv55t for pod openshift-network-diagnostics/network-check-target-qqqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:46.412691 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.412680    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t podName:f5918960-9d76-46ac-8ce5-82c432a9e284 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:46.912661815 +0000 UTC m=+3.038596793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pv55t" (UniqueName: "kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t") pod "network-check-target-qqqj9" (UID: "f5918960-9d76-46ac-8ce5-82c432a9e284") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:46.414226 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.414183    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rmf\" (UniqueName: \"kubernetes.io/projected/b7a2fe62-a684-4216-a5e3-1c1cd0b592b1-kube-api-access-v6rmf\") pod \"tuned-jwzrf\" (UID: \"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1\") " pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.415361 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.415338    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4tr\" (UniqueName: \"kubernetes.io/projected/2bd69c75-0071-4835-a044-9accda5ecee4-kube-api-access-tw4tr\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:46.415452 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.415418    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dzd\" (UniqueName: \"kubernetes.io/projected/482bb2e3-16af-4029-b9cb-4ce88e7cc092-kube-api-access-c6dzd\") pod \"node-ca-tjbks\" (UID: \"482bb2e3-16af-4029-b9cb-4ce88e7cc092\") " pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.415681 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.415660    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldn9\" (UniqueName: \"kubernetes.io/projected/9b2c4b45-013f-4e4a-8e8c-a73b0bba4498-kube-api-access-dldn9\") pod \"multus-additional-cni-plugins-9k5md\" (UID: \"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498\") " pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.416285 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.416230    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6h8\" (UniqueName: \"kubernetes.io/projected/794706ac-dc6b-451e-9dfc-688409907cf9-kube-api-access-5k6h8\") pod \"node-resolver-9jvs2\" (UID: \"794706ac-dc6b-451e-9dfc-688409907cf9\") " pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.494806 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.494765    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj"
Jul 18 00:40:46.502909 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.502878    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:40:46.512115 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.512088    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:40:46.518980 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.518956    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-546f9"
Jul 18 00:40:46.531772 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.531748    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p9wrt"
Jul 18 00:40:46.539275 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.539257    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jvs2"
Jul 18 00:40:46.547475 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.547449    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwzrf"
Jul 18 00:40:46.555152 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.555134    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tjbks"
Jul 18 00:40:46.560922 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.560901    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9k5md"
Jul 18 00:40:46.809038 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.808949    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:46.809184 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.809105    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:46.809221 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.809187    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret podName:a089e7e1-1597-4899-83b2-357a35bb40cb nodeName:}" failed. No retries permitted until 2026-07-18 00:40:47.809158934 +0000 UTC m=+3.935093896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret") pod "global-pull-secret-syncer-tc4gd" (UID: "a089e7e1-1597-4899-83b2-357a35bb40cb") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:46.838651 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.838610    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:46.910049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:46.910018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:46.910193 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.910167    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:46.910257 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:46.910246    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs podName:2bd69c75-0071-4835-a044-9accda5ecee4 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:47.910229244 +0000 UTC m=+4.036164223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs") pod "network-metrics-daemon-4p62b" (UID: "2bd69c75-0071-4835-a044-9accda5ecee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:47.011058 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.011022    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:47.011210 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.011189    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:47.011210 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.011210    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:47.011278 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.011220    2554 projected.go:196] Error preparing data for projected volume kube-api-access-pv55t for pod openshift-network-diagnostics/network-check-target-qqqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:47.011315 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.011282    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t podName:f5918960-9d76-46ac-8ce5-82c432a9e284 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:48.011264924 +0000 UTC m=+4.137199900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pv55t" (UniqueName: "kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t") pod "network-check-target-qqqj9" (UID: "f5918960-9d76-46ac-8ce5-82c432a9e284") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:47.205871 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.205830    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fbcf08_8b1c_476d_9814_a803fe24cfcd.slice/crio-4a1acbef3c32058cac28d2efbc17038cfd396a96256a49a8343fb7ac1d890fe3 WatchSource:0}: Error finding container 4a1acbef3c32058cac28d2efbc17038cfd396a96256a49a8343fb7ac1d890fe3: Status 404 returned error can't find the container with id 4a1acbef3c32058cac28d2efbc17038cfd396a96256a49a8343fb7ac1d890fe3
Jul 18 00:40:47.207393 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.207368    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0621ebeb_5101_4564_9c03_a208a0a105f7.slice/crio-58926d54615e2712ef154334dbbbf324b70c399bf2d586eff39027545f42c071 WatchSource:0}: Error finding container 58926d54615e2712ef154334dbbbf324b70c399bf2d586eff39027545f42c071: Status 404 returned error can't find the container with id 58926d54615e2712ef154334dbbbf324b70c399bf2d586eff39027545f42c071
Jul 18 00:40:47.209053 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.209007    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a32a19_9383_4228_9d56_a207a9ebb4cb.slice/crio-da602aca25397a393ddd9669caa93385b1c0231675b68fce30c9afd5b8c7d721 WatchSource:0}: Error finding container da602aca25397a393ddd9669caa93385b1c0231675b68fce30c9afd5b8c7d721: Status 404 returned error can't find the container with id da602aca25397a393ddd9669caa93385b1c0231675b68fce30c9afd5b8c7d721
Jul 18 00:40:47.211818 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.211792    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761c310d_2fe7_4644_aec4_2182615102af.slice/crio-195a85558fdf49752f8808d7b8d25f43b1bb0735138f9b189b831bdb6a86a3b5 WatchSource:0}: Error finding container 195a85558fdf49752f8808d7b8d25f43b1bb0735138f9b189b831bdb6a86a3b5: Status 404 returned error can't find the container with id 195a85558fdf49752f8808d7b8d25f43b1bb0735138f9b189b831bdb6a86a3b5
Jul 18 00:40:47.212719 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.212682    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a2fe62_a684_4216_a5e3_1c1cd0b592b1.slice/crio-131a0e8d28c53574c9b58df1143515bba4e6f7389e08208132b5f0a0a6c1e9fc WatchSource:0}: Error finding container 131a0e8d28c53574c9b58df1143515bba4e6f7389e08208132b5f0a0a6c1e9fc: Status 404 returned error can't find the container with id 131a0e8d28c53574c9b58df1143515bba4e6f7389e08208132b5f0a0a6c1e9fc
Jul 18 00:40:47.213335 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.213265    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e3ae46_70d9_46c9_b8a1_73bae5d6437b.slice/crio-e2b23aafacd4a6e6734ba21a9974f8af58eb3f0ccce5fe141f2d5a6d5a59cec3 WatchSource:0}: Error finding container e2b23aafacd4a6e6734ba21a9974f8af58eb3f0ccce5fe141f2d5a6d5a59cec3: Status 404 returned error can't find the container with id e2b23aafacd4a6e6734ba21a9974f8af58eb3f0ccce5fe141f2d5a6d5a59cec3
Jul 18 00:40:47.236307 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.236257    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 00:35:46 +0000 UTC" deadline="2028-01-20 11:29:44.159842797 +0000 UTC"
Jul 18 00:40:47.236307 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.236299    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13234h48m56.923546721s"
Jul 18 00:40:47.242340 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.241973    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482bb2e3_16af_4029_b9cb_4ce88e7cc092.slice/crio-abbb2761f21e2552091d4686a9b19b0daa8cdbe099bc43d4d7a9c1d1e0ab6a9e WatchSource:0}: Error finding container abbb2761f21e2552091d4686a9b19b0daa8cdbe099bc43d4d7a9c1d1e0ab6a9e: Status 404 returned error can't find the container with id abbb2761f21e2552091d4686a9b19b0daa8cdbe099bc43d4d7a9c1d1e0ab6a9e
Jul 18 00:40:47.242709 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.242500    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2c4b45_013f_4e4a_8e8c_a73b0bba4498.slice/crio-1634c217949d209cb8990e86ff7df0b4ecf19fb05297ed16530f3709a631e778 WatchSource:0}: Error finding container 1634c217949d209cb8990e86ff7df0b4ecf19fb05297ed16530f3709a631e778: Status 404 returned error can't find the container with id 1634c217949d209cb8990e86ff7df0b4ecf19fb05297ed16530f3709a631e778
Jul 18 00:40:47.243435 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:40:47.243392    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794706ac_dc6b_451e_9dfc_688409907cf9.slice/crio-06369ddd73aacffaaeed0ca47f334edb78bfc7a2b235a5d0bedd9046574a9c7a WatchSource:0}: Error finding container 06369ddd73aacffaaeed0ca47f334edb78bfc7a2b235a5d0bedd9046574a9c7a: Status 404 returned error can't find the container with id 06369ddd73aacffaaeed0ca47f334edb78bfc7a2b235a5d0bedd9046574a9c7a
Jul 18 00:40:47.330695 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.330667    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:47.330797 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.330778    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:40:47.339163 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.339134    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-546f9" event={"ID":"761c310d-2fe7-4644-aec4-2182615102af","Type":"ContainerStarted","Data":"195a85558fdf49752f8808d7b8d25f43b1bb0735138f9b189b831bdb6a86a3b5"}
Jul 18 00:40:47.340435 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.340410    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p9wrt" event={"ID":"97a32a19-9383-4228-9d56-a207a9ebb4cb","Type":"ContainerStarted","Data":"da602aca25397a393ddd9669caa93385b1c0231675b68fce30c9afd5b8c7d721"}
Jul 18 00:40:47.341509 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.341487    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"4a1acbef3c32058cac28d2efbc17038cfd396a96256a49a8343fb7ac1d890fe3"}
Jul 18 00:40:47.342511 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.342486    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jvs2" event={"ID":"794706ac-dc6b-451e-9dfc-688409907cf9","Type":"ContainerStarted","Data":"06369ddd73aacffaaeed0ca47f334edb78bfc7a2b235a5d0bedd9046574a9c7a"}
Jul 18 00:40:47.343665 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.343643    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerStarted","Data":"1634c217949d209cb8990e86ff7df0b4ecf19fb05297ed16530f3709a631e778"}
Jul 18 00:40:47.344774 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.344744    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj" event={"ID":"40e3ae46-70d9-46c9-b8a1-73bae5d6437b","Type":"ContainerStarted","Data":"e2b23aafacd4a6e6734ba21a9974f8af58eb3f0ccce5fe141f2d5a6d5a59cec3"}
Jul 18 00:40:47.346143 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.346119    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t96fg" event={"ID":"0621ebeb-5101-4564-9c03-a208a0a105f7","Type":"ContainerStarted","Data":"58926d54615e2712ef154334dbbbf324b70c399bf2d586eff39027545f42c071"}
Jul 18 00:40:47.348192 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.348155    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal" event={"ID":"6a0ab874ed9de1072b24764a5ad02232","Type":"ContainerStarted","Data":"8b675dedd3b79c70b35ea30e215f14a5d74630ccb2d61a6d630d4b6d510e2f57"}
Jul 18 00:40:47.349291 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.349270    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tjbks" event={"ID":"482bb2e3-16af-4029-b9cb-4ce88e7cc092","Type":"ContainerStarted","Data":"abbb2761f21e2552091d4686a9b19b0daa8cdbe099bc43d4d7a9c1d1e0ab6a9e"}
Jul 18 00:40:47.350347 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.350326    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwzrf" event={"ID":"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1","Type":"ContainerStarted","Data":"131a0e8d28c53574c9b58df1143515bba4e6f7389e08208132b5f0a0a6c1e9fc"}
Jul 18 00:40:47.361234 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.361188    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-161.ec2.internal" podStartSLOduration=2.361171086 podStartE2EDuration="2.361171086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:47.360723116 +0000 UTC m=+3.486658096" watchObservedRunningTime="2026-07-18 00:40:47.361171086 +0000 UTC m=+3.487106065"
Jul 18 00:40:47.817358 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.816948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:47.817522 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.817365    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:47.817522 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.817424    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret podName:a089e7e1-1597-4899-83b2-357a35bb40cb nodeName:}" failed. No retries permitted until 2026-07-18 00:40:49.817408012 +0000 UTC m=+5.943342969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret") pod "global-pull-secret-syncer-tc4gd" (UID: "a089e7e1-1597-4899-83b2-357a35bb40cb") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:47.918665 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:47.918325    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:47.918665 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.918468    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:47.918665 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:47.918530    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs podName:2bd69c75-0071-4835-a044-9accda5ecee4 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:49.918512575 +0000 UTC m=+6.044447535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs") pod "network-metrics-daemon-4p62b" (UID: "2bd69c75-0071-4835-a044-9accda5ecee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:48.019761 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:48.019648    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:48.019940 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:48.019812    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:48.019940 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:48.019832    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:48.019940 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:48.019844    2554 projected.go:196] Error preparing data for projected volume kube-api-access-pv55t for pod openshift-network-diagnostics/network-check-target-qqqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:48.019940 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:48.019899    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t podName:f5918960-9d76-46ac-8ce5-82c432a9e284 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:50.01988211 +0000 UTC m=+6.145817067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pv55t" (UniqueName: "kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t") pod "network-check-target-qqqj9" (UID: "f5918960-9d76-46ac-8ce5-82c432a9e284") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:48.237345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:48.237296    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 00:35:46 +0000 UTC" deadline="2028-02-13 08:24:53.329871082 +0000 UTC"
Jul 18 00:40:48.237345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:48.237335    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13807h44m5.092540225s"
Jul 18 00:40:48.332540 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:48.332511    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:48.332735 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:48.332658    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:40:48.333095 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:48.333075    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:48.333211 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:48.333191    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:40:48.385268 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:48.384007    2554 generic.go:356] "Generic (PLEG): container finished" podID="8cf2fde091123b7b6cb170fddd088984" containerID="e8c600fd00e1e8d40542cc750e5e9267b2716f19bac5ae87864d40b800e0fc69" exitCode=0
Jul 18 00:40:48.385268 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:48.384988    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal" event={"ID":"8cf2fde091123b7b6cb170fddd088984","Type":"ContainerDied","Data":"e8c600fd00e1e8d40542cc750e5e9267b2716f19bac5ae87864d40b800e0fc69"}
Jul 18 00:40:49.332103 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:49.330308    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:49.332103 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:49.330436    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:40:49.416379 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:49.416342    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal" event={"ID":"8cf2fde091123b7b6cb170fddd088984","Type":"ContainerStarted","Data":"8d90344ab96d830fdc542670f29a9c16eacd9be8aced32b05f16a320dcb7c052"}
Jul 18 00:40:49.835592 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:49.835555    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:49.835773 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:49.835747    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:49.835843 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:49.835812    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret podName:a089e7e1-1597-4899-83b2-357a35bb40cb nodeName:}" failed. No retries permitted until 2026-07-18 00:40:53.835793801 +0000 UTC m=+9.961728768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret") pod "global-pull-secret-syncer-tc4gd" (UID: "a089e7e1-1597-4899-83b2-357a35bb40cb") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:49.936924 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:49.936886    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:49.937533 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:49.937090    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:49.937533 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:49.937161    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs podName:2bd69c75-0071-4835-a044-9accda5ecee4 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:53.937142051 +0000 UTC m=+10.063077015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs") pod "network-metrics-daemon-4p62b" (UID: "2bd69c75-0071-4835-a044-9accda5ecee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:50.038185 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:50.037549    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:50.038185 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:50.037736    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:50.038185 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:50.037762    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:50.038185 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:50.037784    2554 projected.go:196] Error preparing data for projected volume kube-api-access-pv55t for pod openshift-network-diagnostics/network-check-target-qqqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:50.038185 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:50.037843    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t podName:f5918960-9d76-46ac-8ce5-82c432a9e284 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:54.037824056 +0000 UTC m=+10.163759015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pv55t" (UniqueName: "kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t") pod "network-check-target-qqqj9" (UID: "f5918960-9d76-46ac-8ce5-82c432a9e284") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:50.330808 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:50.330046    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:50.330808 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:50.330194    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:40:50.330808 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:50.330604    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:50.330808 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:50.330711    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:40:51.331264 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:51.330759    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:51.331264 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:51.330933    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:40:52.330325 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:52.330250    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:52.330325 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:52.330324    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:52.330555 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:52.330464    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:40:52.330627 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:52.330574    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:40:53.330275 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:53.330230    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:53.330789 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:53.330382    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:40:53.871876 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:53.871835    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:53.872063 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:53.872002    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:53.872120 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:53.872068    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret podName:a089e7e1-1597-4899-83b2-357a35bb40cb nodeName:}" failed. No retries permitted until 2026-07-18 00:41:01.872052028 +0000 UTC m=+17.997986984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret") pod "global-pull-secret-syncer-tc4gd" (UID: "a089e7e1-1597-4899-83b2-357a35bb40cb") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:53.972445 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:53.972407    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:53.972652 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:53.972549    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:53.972724 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:53.972700    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs podName:2bd69c75-0071-4835-a044-9accda5ecee4 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:01.972603131 +0000 UTC m=+18.098538104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs") pod "network-metrics-daemon-4p62b" (UID: "2bd69c75-0071-4835-a044-9accda5ecee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:54.073322 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:54.073157    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:54.073502 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:54.073390    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:54.073502 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:54.073411    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:54.073502 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:54.073423    2554 projected.go:196] Error preparing data for projected volume kube-api-access-pv55t for pod openshift-network-diagnostics/network-check-target-qqqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:54.073502 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:54.073483    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t podName:f5918960-9d76-46ac-8ce5-82c432a9e284 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:02.073465956 +0000 UTC m=+18.199400934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pv55t" (UniqueName: "kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t") pod "network-check-target-qqqj9" (UID: "f5918960-9d76-46ac-8ce5-82c432a9e284") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:54.331021 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:54.330873    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:54.331021 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:54.330997    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:40:54.331513 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:54.331356    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:54.331513 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:54.331458    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:40:55.330361 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:55.330325    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:55.330533 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:55.330475    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:40:56.330432 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:56.330397    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:56.330912 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:56.330534    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:40:56.330912 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:56.330686    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:56.330912 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:56.330773    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:40:57.330671 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:57.330637    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:57.331117 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:57.330768    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:40:58.330527 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:58.330490    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:40:58.330748 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:58.330535    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:40:58.330748 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:58.330639    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:40:58.331104 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:58.330757    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:40:59.330481 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:40:59.330446    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:40:59.330676 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:40:59.330565    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:00.333584 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:00.333553    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:00.334031 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:00.333702    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:00.334031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:00.333755    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:00.334031 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:00.333865    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:01.330644 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:01.330593    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:01.330846 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:01.330748    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:01.935036 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:01.934943    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:01.935506 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:01.935085    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:41:01.935506 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:01.935170    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret podName:a089e7e1-1597-4899-83b2-357a35bb40cb nodeName:}" failed. No retries permitted until 2026-07-18 00:41:17.935147061 +0000 UTC m=+34.061082023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret") pod "global-pull-secret-syncer-tc4gd" (UID: "a089e7e1-1597-4899-83b2-357a35bb40cb") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:41:02.035921 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:02.035887    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:02.036115 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.036036    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:41:02.036115 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.036108    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs podName:2bd69c75-0071-4835-a044-9accda5ecee4 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:18.036083803 +0000 UTC m=+34.162018761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs") pod "network-metrics-daemon-4p62b" (UID: "2bd69c75-0071-4835-a044-9accda5ecee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:41:02.137258 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:02.137222    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:02.137432 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.137382    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:41:02.137432 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.137400    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:41:02.137432 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.137414    2554 projected.go:196] Error preparing data for projected volume kube-api-access-pv55t for pod openshift-network-diagnostics/network-check-target-qqqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:41:02.137566 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.137471    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t podName:f5918960-9d76-46ac-8ce5-82c432a9e284 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:18.137453998 +0000 UTC m=+34.263388966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pv55t" (UniqueName: "kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t") pod "network-check-target-qqqj9" (UID: "f5918960-9d76-46ac-8ce5-82c432a9e284") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:41:02.330540 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:02.330465    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:02.330728 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.330594    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:02.330728 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:02.330662    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:02.330897 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:02.330778    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:03.330277 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:03.330242    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:03.330686 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:03.330364    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:04.331246 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:04.331214    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:04.331745 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:04.331342    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:04.331745 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:04.331404    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:04.331745 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:04.331513    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:05.330909 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:05.330865    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:05.331087 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:05.331008    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:06.333130 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.332922    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:06.333130 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:06.333037    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:06.333582 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.333223    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:06.334554 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:06.334506    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:06.452248 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.452090    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tjbks" event={"ID":"482bb2e3-16af-4029-b9cb-4ce88e7cc092","Type":"ContainerStarted","Data":"1ea8afb9e3a14bad369c0c76cb60949fdfaed4f8e112bbe683d551dbb0dfdb4e"}
Jul 18 00:41:06.453467 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.453447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwzrf" event={"ID":"b7a2fe62-a684-4216-a5e3-1c1cd0b592b1","Type":"ContainerStarted","Data":"824b54cc45c8d98e3882850ba70d13dc4a33cf8859249dc5212e72f3c499528e"}
Jul 18 00:41:06.455227 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.455183    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-546f9" event={"ID":"761c310d-2fe7-4644-aec4-2182615102af","Type":"ContainerStarted","Data":"cbde8adc115f1c685918c57ea35c81a48d430c28a4d348092b90570df8d7430d"}
Jul 18 00:41:06.457333 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.457315    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"0f8019f6030dc03dcba60289ab6d055e6893b7bf35c1b18e2dc73ebb8983c209"}
Jul 18 00:41:06.457440 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.457374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"2f56a8d1d6997f096515cd62c47921d5777522c7ad146cdef0f4882bf50a832e"}
Jul 18 00:41:06.458794 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.458771    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jvs2" event={"ID":"794706ac-dc6b-451e-9dfc-688409907cf9","Type":"ContainerStarted","Data":"99567d0aa847290556c579b01cf13f183a6ea111914ec3c744635e2ab8b3459e"}
Jul 18 00:41:06.465152 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.465067    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b2c4b45-013f-4e4a-8e8c-a73b0bba4498" containerID="c98a0800491c8e4776ffd1890327c633412e33d6c0b1a62033d79d4c437344dd" exitCode=0
Jul 18 00:41:06.465272 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.465149    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerDied","Data":"c98a0800491c8e4776ffd1890327c633412e33d6c0b1a62033d79d4c437344dd"}
Jul 18 00:41:06.466495 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.466288    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tjbks" podStartSLOduration=3.560743511 podStartE2EDuration="22.466275406s" totalImagesPullingTime="18.905531895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:06.466073483 +0000 UTC m=+22.592008457" watchObservedRunningTime="2026-07-18 00:41:06.466275406 +0000 UTC m=+22.592210443"
Jul 18 00:41:06.466495 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.466354    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-161.ec2.internal" podStartSLOduration=21.466350148 podStartE2EDuration="21.466350148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:49.43271415 +0000 UTC m=+5.558649130" watchObservedRunningTime="2026-07-18 00:41:06.466350148 +0000 UTC m=+22.592285127"
Jul 18 00:41:06.466956 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.466820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj" event={"ID":"40e3ae46-70d9-46c9-b8a1-73bae5d6437b","Type":"ContainerStarted","Data":"0e8b05d8b4f0b64a8ab1151179a8c9ca2a2525c4b3109515b9244bc3356e7a45"}
Jul 18 00:41:06.468093 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.468077    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t96fg" event={"ID":"0621ebeb-5101-4564-9c03-a208a0a105f7","Type":"ContainerStarted","Data":"6e532e760f1ebca6f061865a1c66433088dffcab7f2160bf67253d06c78d184d"}
Jul 18 00:41:06.503778 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.503735    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-546f9" podStartSLOduration=3.527290577 podStartE2EDuration="22.503719248s" totalImagesPullingTime="18.976428671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:06.481347817 +0000 UTC m=+22.607282797" watchObservedRunningTime="2026-07-18 00:41:06.503719248 +0000 UTC m=+22.629654226"
Jul 18 00:41:06.515987 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.515887    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9jvs2" podStartSLOduration=3.580976641 podStartE2EDuration="22.51587142s" totalImagesPullingTime="18.934894779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:06.515683636 +0000 UTC m=+22.641618661" watchObservedRunningTime="2026-07-18 00:41:06.51587142 +0000 UTC m=+22.641806399"
Jul 18 00:41:06.533153 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.533102    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jwzrf" podStartSLOduration=3.639619835 podStartE2EDuration="22.533086936s" totalImagesPullingTime="18.893467101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:06.532870542 +0000 UTC m=+22.658805522" watchObservedRunningTime="2026-07-18 00:41:06.533086936 +0000 UTC m=+22.659021912"
Jul 18 00:41:06.549199 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:06.549154    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t96fg" podStartSLOduration=3.625457373 podStartE2EDuration="22.54913764s" totalImagesPullingTime="18.923680267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:06.548533375 +0000 UTC m=+22.674468357" watchObservedRunningTime="2026-07-18 00:41:06.54913764 +0000 UTC m=+22.675072619"
Jul 18 00:41:07.330831 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.330794    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:07.330999 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:07.330924    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:07.474112 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.474028    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"7e234d7962ad46e54ac7dbf02945ef4eabdf9c418bc703580b8bfe71d3047bcc"}
Jul 18 00:41:07.475089 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.474127    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"0f9b7c9f47f499907bb5112bc55a4b6c5b1605bd70b57758c650d2273ef93d4b"}
Jul 18 00:41:07.475089 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.474174    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"e467967bd22afd6a3a65ef3a5360e6083b9d4d0bb3fef025b558c9f281babb28"}
Jul 18 00:41:07.475089 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.474222    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"d85159593e0f216c9ffbf74938f6f577f2224de7c7faf4cbbf9a1f86e580811a"}
Jul 18 00:41:07.666072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.666047    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:41:07.666687 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.666665    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:41:07.718167 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:07.718129    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 18 00:41:08.260779 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.260669    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T00:41:07.718160009Z","UUID":"a2e7bc59-5ee5-4cfa-89ec-5168d19a5b1f","Handler":null,"Name":"","Endpoint":""}
Jul 18 00:41:08.262391 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.262369    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 18 00:41:08.262539 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.262412    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 18 00:41:08.331167 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.330898    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:08.331167 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.330900    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:08.331167 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:08.331012    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:08.331167 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:08.331084    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:08.479482 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.479440    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj" event={"ID":"40e3ae46-70d9-46c9-b8a1-73bae5d6437b","Type":"ContainerStarted","Data":"a8237bc4f96f52a8b4fb0b88e28ac6819727bd587f7c54ffe3d869a6621f98ed"}
Jul 18 00:41:08.481403 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.481347    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p9wrt" event={"ID":"97a32a19-9383-4228-9d56-a207a9ebb4cb","Type":"ContainerStarted","Data":"b52f03c049dbc3f4059905f787b756bf58d8b7f2904649264379727248d3bcd6"}
Jul 18 00:41:08.481803 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.481781    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:41:08.482250 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.482229    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t96fg"
Jul 18 00:41:08.495558 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:08.495505    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p9wrt" podStartSLOduration=5.572450612 podStartE2EDuration="24.495485943s" totalImagesPullingTime="18.923035331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:08.494517712 +0000 UTC m=+24.620452693" watchObservedRunningTime="2026-07-18 00:41:08.495485943 +0000 UTC m=+24.621420924"
Jul 18 00:41:09.330456 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:09.330420    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:09.330674 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:09.330534    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:09.486097 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:09.486046    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj" event={"ID":"40e3ae46-70d9-46c9-b8a1-73bae5d6437b","Type":"ContainerStarted","Data":"ebbbff5c960280ca3330bcb9921df832a29ed9a7a8f3c79d3fd6f1dfdecc36d3"}
Jul 18 00:41:09.503388 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:09.503327    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fggvj" podStartSLOduration=4.452493575 podStartE2EDuration="25.503306347s" totalImagesPullingTime="21.050812772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:09.503216085 +0000 UTC m=+25.629151059" watchObservedRunningTime="2026-07-18 00:41:09.503306347 +0000 UTC m=+25.629241325"
Jul 18 00:41:10.330413 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:10.330225    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:10.330590 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:10.330290    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:10.330590 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:10.330523    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:10.330716 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:10.330602    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:11.330405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:11.330370    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:11.331051 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:11.330498    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:11.493686 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:11.493650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"f2372c9d474a87bc84031ff2696acf94dbbc87d724f950d7df6db234161e33ad"}
Jul 18 00:41:11.495217 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:11.495191    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b2c4b45-013f-4e4a-8e8c-a73b0bba4498" containerID="3b733f7f0a4beb98d089aa0dce9ed4653de8280de3a83478c166144df42c64e5" exitCode=0
Jul 18 00:41:11.495335 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:11.495223    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerDied","Data":"3b733f7f0a4beb98d089aa0dce9ed4653de8280de3a83478c166144df42c64e5"}
Jul 18 00:41:12.330832 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:12.330797    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:12.330832 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:12.330797    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:12.331271 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:12.330948    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:12.331271 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:12.331081    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:13.330364 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:13.330204    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:13.330523 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:13.330444    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:13.502192 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:13.502157    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" event={"ID":"05fbcf08-8b1c-476d-9814-a803fe24cfcd","Type":"ContainerStarted","Data":"219da28455e606cff15dda60389d9d52827b4d390dc84beee187301084a0fea9"}
Jul 18 00:41:13.502859 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:13.502469    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:41:13.504403 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:13.504372    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b2c4b45-013f-4e4a-8e8c-a73b0bba4498" containerID="829fa2c41f84de5a88df39e5a5e2b7a9ac740baeb6eaa7cb1590500eda650cfa" exitCode=0
Jul 18 00:41:13.504514 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:13.504408    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerDied","Data":"829fa2c41f84de5a88df39e5a5e2b7a9ac740baeb6eaa7cb1590500eda650cfa"}
Jul 18 00:41:13.518807 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:13.518782    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:41:13.531254 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:13.531215    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4" podStartSLOduration=10.489510133 podStartE2EDuration="29.531201838s" totalImagesPullingTime="19.041691705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:13.529922839 +0000 UTC m=+29.655857841" watchObservedRunningTime="2026-07-18 00:41:13.531201838 +0000 UTC m=+29.657136817"
Jul 18 00:41:14.333593 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.333563    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:14.333761 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:14.333681    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:14.333761 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.333733    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:14.333879 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:14.333841    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:14.507584 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.507551    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:41:14.508046 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.507660    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:41:14.523605 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.523575    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:41:14.795173 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.794611    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qqqj9"]
Jul 18 00:41:14.795173 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.794749    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:14.795173 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:14.794847    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:14.795401 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.795204    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4p62b"]
Jul 18 00:41:14.795401 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.795306    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:14.795507 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:14.795423    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:14.795800 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.795758    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tc4gd"]
Jul 18 00:41:14.795882 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:14.795847    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:14.795930 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:14.795917    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:15.513063 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:15.512846    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b2c4b45-013f-4e4a-8e8c-a73b0bba4498" containerID="a21ea83f52ef9d89e87c174684a24595bdb34d824a18b2d78595eff4a600c65a" exitCode=0
Jul 18 00:41:15.513529 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:15.512935    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerDied","Data":"a21ea83f52ef9d89e87c174684a24595bdb34d824a18b2d78595eff4a600c65a"}
Jul 18 00:41:16.330779 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:16.330743    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:16.330993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:16.330815    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:16.330993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:16.330832    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:16.330993 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:16.330927    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:16.330993 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:16.330975    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:16.331205 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:16.331077    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:17.960633 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:17.960528    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:17.961150 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:17.960723    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:41:17.961150 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:17.960806    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret podName:a089e7e1-1597-4899-83b2-357a35bb40cb nodeName:}" failed. No retries permitted until 2026-07-18 00:41:49.960782848 +0000 UTC m=+66.086717826 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret") pod "global-pull-secret-syncer-tc4gd" (UID: "a089e7e1-1597-4899-83b2-357a35bb40cb") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:41:18.061171 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:18.061127    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:18.061343 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.061298    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:41:18.061419 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.061387    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs podName:2bd69c75-0071-4835-a044-9accda5ecee4 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:50.06136452 +0000 UTC m=+66.187299477 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs") pod "network-metrics-daemon-4p62b" (UID: "2bd69c75-0071-4835-a044-9accda5ecee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:41:18.162511 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:18.162472    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:18.162731 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.162690    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:41:18.162731 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.162713    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:41:18.162731 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.162727    2554 projected.go:196] Error preparing data for projected volume kube-api-access-pv55t for pod openshift-network-diagnostics/network-check-target-qqqj9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:41:18.162892 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.162794    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t podName:f5918960-9d76-46ac-8ce5-82c432a9e284 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:50.162776096 +0000 UTC m=+66.288711078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pv55t" (UniqueName: "kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t") pod "network-check-target-qqqj9" (UID: "f5918960-9d76-46ac-8ce5-82c432a9e284") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:41:18.178839 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:18.178815    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9jvs2_794706ac-dc6b-451e-9dfc-688409907cf9/dns-node-resolver/0.log"
Jul 18 00:41:18.330261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:18.330223    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:18.330427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:18.330235    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:18.330427 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.330361    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:18.330427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:18.330378    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:18.330591 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.330490    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:18.330591 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:18.330566    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:19.354522 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:19.354490    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tjbks_482bb2e3-16af-4029-b9cb-4ce88e7cc092/node-ca/0.log"
Jul 18 00:41:20.330846 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:20.330812    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:20.330846 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:20.330841    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:20.331049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:20.330812    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:20.331049 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:20.330922    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:20.331049 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:20.330982    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:20.331147 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:20.331076    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:20.527608 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:20.527575    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b2c4b45-013f-4e4a-8e8c-a73b0bba4498" containerID="937adf41a50e5fa3f0e3dad3c36cd8a38c18a6e0b5a471f60ce20dce1dea5bbe" exitCode=0
Jul 18 00:41:20.528079 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:20.527633    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerDied","Data":"937adf41a50e5fa3f0e3dad3c36cd8a38c18a6e0b5a471f60ce20dce1dea5bbe"}
Jul 18 00:41:21.532215 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:21.532178    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b2c4b45-013f-4e4a-8e8c-a73b0bba4498" containerID="0ba9cf3ba76f89dddf1e66d8b2c715ebe12872c0e3a66732e80490341b786a46" exitCode=0
Jul 18 00:41:21.532715 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:21.532226    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerDied","Data":"0ba9cf3ba76f89dddf1e66d8b2c715ebe12872c0e3a66732e80490341b786a46"}
Jul 18 00:41:22.330553 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:22.330519    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:22.330553 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:22.330550    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:22.330889 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:22.330519    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:22.330889 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:22.330672    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:22.330889 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:22.330727    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:22.330889 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:22.330796    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:22.538263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:22.538228    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9k5md" event={"ID":"9b2c4b45-013f-4e4a-8e8c-a73b0bba4498","Type":"ContainerStarted","Data":"5d65f589eb01cd90ece00c1e38ecc7694adfc4978bca8b5eca1ca8d63c08c489"}
Jul 18 00:41:22.566331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:22.566281    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9k5md" podStartSLOduration=8.974248429 podStartE2EDuration="38.566264855s" totalImagesPullingTime="29.592016426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:22.565713576 +0000 UTC m=+38.691648557" watchObservedRunningTime="2026-07-18 00:41:22.566264855 +0000 UTC m=+38.692199834"
Jul 18 00:41:24.333659 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:24.333606    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:24.334027 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:24.333711    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:24.334027 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:24.333730    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:24.334027 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:24.333741    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:24.334027 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:24.333809    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:24.334027 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:24.333866    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:26.330000 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:26.329967    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:26.330461 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:26.329967    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:26.330461 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:26.330082    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:26.330461 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:26.329972    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:26.330461 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:26.330179    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:26.330461 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:26.330305    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:28.330560 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:28.330525    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:28.330983 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:28.330525    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:28.330983 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:28.330659    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:28.330983 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:28.330525    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:28.330983 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:28.330731    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:28.330983 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:28.330777    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:30.330721 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:30.330684    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:30.331096 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:30.330684    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:30.331096 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:30.330810    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:30.331096 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:30.330864    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:30.331096 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:30.330688    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:30.331096 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:30.330928    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:32.330889 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:32.330852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:32.331391 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:32.330898    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:32.331391 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:32.331010    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:32.331391 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:32.331050    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:32.331391 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:32.331110    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:32.331391 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:32.331166    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:34.331083 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:34.331051    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:34.331476 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:34.331134    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:34.331476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:34.331230    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:34.331476 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:34.331326    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:34.331476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:34.331399    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:34.331476 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:34.331451    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:36.329955 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:36.329920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:36.330491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:36.329929    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:36.330491 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:36.330059    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:36.330491 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:36.329929    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:36.330491 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:36.330097    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:36.330491 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:36.330179    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:38.330842 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:38.330806    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:38.331303 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:38.330811    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:38.331303 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:38.330917    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:38.331303 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:38.330810    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:38.331303 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:38.330990    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:38.331303 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:38.331059    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:40.330285 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:40.330242    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:40.330285 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:40.330280    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:40.330837 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:40.330251    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:40.330837 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:40.330352    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc4gd" podUID="a089e7e1-1597-4899-83b2-357a35bb40cb"
Jul 18 00:41:40.330837 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:40.330458    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqqj9" podUID="f5918960-9d76-46ac-8ce5-82c432a9e284"
Jul 18 00:41:40.330837 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:40.330530    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4p62b" podUID="2bd69c75-0071-4835-a044-9accda5ecee4"
Jul 18 00:41:41.212318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.212291    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-161.ec2.internal" event="NodeReady"
Jul 18 00:41:41.212516 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.212406    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 18 00:41:41.257574 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.257536    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6tccd"]
Jul 18 00:41:41.261077 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.260314    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.262536 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.262510    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k28lr"]
Jul 18 00:41:41.263550 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.263529    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 18 00:41:41.263681 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.263662    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cqrgt\""
Jul 18 00:41:41.263752 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.263740    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 18 00:41:41.263800 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.263754    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 18 00:41:41.263800 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.263770    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 18 00:41:41.264577 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.264559    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.266884 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.266865    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 18 00:41:41.266977 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.266910    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 18 00:41:41.267040 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.267007    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 18 00:41:41.267096 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.267061    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h5lw6\""
Jul 18 00:41:41.272524 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.272500    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6tccd"]
Jul 18 00:41:41.291367 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.291342    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k28lr"]
Jul 18 00:41:41.335151 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.335109    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.335151 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.335151    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d36b8cf-0af7-46ad-a1d3-b4456fbba020-cert\") pod \"ingress-canary-k28lr\" (UID: \"9d36b8cf-0af7-46ad-a1d3-b4456fbba020\") " pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.335605 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.335176    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-crio-socket\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.335605 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.335255    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswwz\" (UniqueName: \"kubernetes.io/projected/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-kube-api-access-nswwz\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.335605 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.335283    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.335605 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.335321    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-data-volume\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.335605 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.335352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8bt\" (UniqueName: \"kubernetes.io/projected/9d36b8cf-0af7-46ad-a1d3-b4456fbba020-kube-api-access-8c8bt\") pod \"ingress-canary-k28lr\" (UID: \"9d36b8cf-0af7-46ad-a1d3-b4456fbba020\") " pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.364861 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.364822    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kw5b5"]
Jul 18 00:41:41.366638 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.366601    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.370321 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.370301    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 18 00:41:41.370447 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.370344    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7vqp\""
Jul 18 00:41:41.370715 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.370701    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 18 00:41:41.383428 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.383402    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kw5b5"]
Jul 18 00:41:41.435881 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.435842    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d36b8cf-0af7-46ad-a1d3-b4456fbba020-cert\") pod \"ingress-canary-k28lr\" (UID: \"9d36b8cf-0af7-46ad-a1d3-b4456fbba020\") " pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.435881 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.435883    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-crio-socket\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.436136 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.435958    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-crio-socket\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.436136 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436007    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswwz\" (UniqueName: \"kubernetes.io/projected/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-kube-api-access-nswwz\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.436136 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436043    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b63d0c20-aae3-43de-a175-20b8fb23fd1c-metrics-tls\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.436136 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.436136 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436103    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-data-volume\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.436136 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436118    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8bt\" (UniqueName: \"kubernetes.io/projected/9d36b8cf-0af7-46ad-a1d3-b4456fbba020-kube-api-access-8c8bt\") pod \"ingress-canary-k28lr\" (UID: \"9d36b8cf-0af7-46ad-a1d3-b4456fbba020\") " pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.436136 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436142    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63d0c20-aae3-43de-a175-20b8fb23fd1c-tmp-dir\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.436432 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436188    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.436432 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436213    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b63d0c20-aae3-43de-a175-20b8fb23fd1c-config-volume\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.436432 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5kj\" (UniqueName: \"kubernetes.io/projected/b63d0c20-aae3-43de-a175-20b8fb23fd1c-kube-api-access-7b5kj\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.436568 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436540    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-data-volume\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.436841 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.436824    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.439935 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.439912    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.440042 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.439968    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d36b8cf-0af7-46ad-a1d3-b4456fbba020-cert\") pod \"ingress-canary-k28lr\" (UID: \"9d36b8cf-0af7-46ad-a1d3-b4456fbba020\") " pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.444274 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.444247    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8bt\" (UniqueName: \"kubernetes.io/projected/9d36b8cf-0af7-46ad-a1d3-b4456fbba020-kube-api-access-8c8bt\") pod \"ingress-canary-k28lr\" (UID: \"9d36b8cf-0af7-46ad-a1d3-b4456fbba020\") " pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.444558 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.444532    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswwz\" (UniqueName: \"kubernetes.io/projected/55e9d85e-ba32-4a40-b6d5-b9931ecbd044-kube-api-access-nswwz\") pod \"insights-runtime-extractor-6tccd\" (UID: \"55e9d85e-ba32-4a40-b6d5-b9931ecbd044\") " pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.537102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.537065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b63d0c20-aae3-43de-a175-20b8fb23fd1c-metrics-tls\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.537256 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.537127    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63d0c20-aae3-43de-a175-20b8fb23fd1c-tmp-dir\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.537256 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.537199    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b63d0c20-aae3-43de-a175-20b8fb23fd1c-config-volume\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.537256 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.537225    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5kj\" (UniqueName: \"kubernetes.io/projected/b63d0c20-aae3-43de-a175-20b8fb23fd1c-kube-api-access-7b5kj\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.537657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.537639    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63d0c20-aae3-43de-a175-20b8fb23fd1c-tmp-dir\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.537854 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.537838    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b63d0c20-aae3-43de-a175-20b8fb23fd1c-config-volume\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.539379 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.539358    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b63d0c20-aae3-43de-a175-20b8fb23fd1c-metrics-tls\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.555958 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.555927    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5kj\" (UniqueName: \"kubernetes.io/projected/b63d0c20-aae3-43de-a175-20b8fb23fd1c-kube-api-access-7b5kj\") pod \"dns-default-kw5b5\" (UID: \"b63d0c20-aae3-43de-a175-20b8fb23fd1c\") " pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.571436 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.571412    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6tccd"
Jul 18 00:41:41.576796 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.576772    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k28lr"
Jul 18 00:41:41.674718 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.674685    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:41.714939 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.714907    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k28lr"]
Jul 18 00:41:41.719697 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:41.719657    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d36b8cf_0af7_46ad_a1d3_b4456fbba020.slice/crio-8d1f06c1a7c446cd77a2c007e8228151b17bdb29e4e8b1848f9ac84b505566e5 WatchSource:0}: Error finding container 8d1f06c1a7c446cd77a2c007e8228151b17bdb29e4e8b1848f9ac84b505566e5: Status 404 returned error can't find the container with id 8d1f06c1a7c446cd77a2c007e8228151b17bdb29e4e8b1848f9ac84b505566e5
Jul 18 00:41:41.734176 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.734095    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6tccd"]
Jul 18 00:41:41.738960 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:41.738439    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e9d85e_ba32_4a40_b6d5_b9931ecbd044.slice/crio-929371e46286de5ca016ce010c548a481e623cf65f84fc0f01ae8bd2f6005bc4 WatchSource:0}: Error finding container 929371e46286de5ca016ce010c548a481e623cf65f84fc0f01ae8bd2f6005bc4: Status 404 returned error can't find the container with id 929371e46286de5ca016ce010c548a481e623cf65f84fc0f01ae8bd2f6005bc4
Jul 18 00:41:41.810808 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:41.810703    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kw5b5"]
Jul 18 00:41:41.815540 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:41.814184    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63d0c20_aae3_43de_a175_20b8fb23fd1c.slice/crio-085dda33de4e22a562e3587e12f8be1b3ee8e7f34d59266e634c82c95d5ce6ab WatchSource:0}: Error finding container 085dda33de4e22a562e3587e12f8be1b3ee8e7f34d59266e634c82c95d5ce6ab: Status 404 returned error can't find the container with id 085dda33de4e22a562e3587e12f8be1b3ee8e7f34d59266e634c82c95d5ce6ab
Jul 18 00:41:42.331431 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.331398    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:42.331596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.331398    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:42.331596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.331552    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:42.334299 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.334276    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 18 00:41:42.335892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.335267    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 18 00:41:42.335892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.335666    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9tdp4\""
Jul 18 00:41:42.335892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.335705    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2stqj\""
Jul 18 00:41:42.335892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.335726    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 18 00:41:42.336401 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.335926    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 18 00:41:42.584459 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.584370    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5b5" event={"ID":"b63d0c20-aae3-43de-a175-20b8fb23fd1c","Type":"ContainerStarted","Data":"085dda33de4e22a562e3587e12f8be1b3ee8e7f34d59266e634c82c95d5ce6ab"}
Jul 18 00:41:42.585950 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.585920    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k28lr" event={"ID":"9d36b8cf-0af7-46ad-a1d3-b4456fbba020","Type":"ContainerStarted","Data":"8d1f06c1a7c446cd77a2c007e8228151b17bdb29e4e8b1848f9ac84b505566e5"}
Jul 18 00:41:42.587346 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:42.587309    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tccd" event={"ID":"55e9d85e-ba32-4a40-b6d5-b9931ecbd044","Type":"ContainerStarted","Data":"929371e46286de5ca016ce010c548a481e623cf65f84fc0f01ae8bd2f6005bc4"}
Jul 18 00:41:46.526728 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.526694    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4s4"
Jul 18 00:41:46.642779 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.642742    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"]
Jul 18 00:41:46.646138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.646119    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:46.648602 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.648584    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 18 00:41:46.649112 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.649097    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6m9pt\""
Jul 18 00:41:46.655763 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.655738    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"]
Jul 18 00:41:46.683571 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.683539    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/069929f9-0ece-4af0-a944-5cba5faaf972-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v5swd\" (UID: \"069929f9-0ece-4af0-a944-5cba5faaf972\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:46.784824 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:46.784740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/069929f9-0ece-4af0-a944-5cba5faaf972-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v5swd\" (UID: \"069929f9-0ece-4af0-a944-5cba5faaf972\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:46.784960 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:46.784881    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 18 00:41:46.785007 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:46.784978    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/069929f9-0ece-4af0-a944-5cba5faaf972-tls-certificates podName:069929f9-0ece-4af0-a944-5cba5faaf972 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:47.284954939 +0000 UTC m=+63.410889908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/069929f9-0ece-4af0-a944-5cba5faaf972-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-v5swd" (UID: "069929f9-0ece-4af0-a944-5cba5faaf972") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 18 00:41:47.294404 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.294370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/069929f9-0ece-4af0-a944-5cba5faaf972-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v5swd\" (UID: \"069929f9-0ece-4af0-a944-5cba5faaf972\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:47.297189 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.297154    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/069929f9-0ece-4af0-a944-5cba5faaf972-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v5swd\" (UID: \"069929f9-0ece-4af0-a944-5cba5faaf972\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:47.552860 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.552784    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c4fdd577b-f4x9g"]
Jul 18 00:41:47.554695 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.554635    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:47.555526 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.555511    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.558165 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.558144    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 18 00:41:47.558956 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.558940    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 18 00:41:47.559228 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.559206    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 18 00:41:47.559310 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.559294    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 18 00:41:47.559380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.559295    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 18 00:41:47.559448 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.559428    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 18 00:41:47.559448 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.559438    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 18 00:41:47.559658 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.559605    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-56hhc\""
Jul 18 00:41:47.565584 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.565517    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4fdd577b-f4x9g"]
Jul 18 00:41:47.604953 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.604452    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5b5" event={"ID":"b63d0c20-aae3-43de-a175-20b8fb23fd1c","Type":"ContainerStarted","Data":"c647998f16f0fe1eba9bb2d732ffe53b3e712ab7ab67bdcadd18e1d2c415a478"}
Jul 18 00:41:47.604953 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.604513    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kw5b5" event={"ID":"b63d0c20-aae3-43de-a175-20b8fb23fd1c","Type":"ContainerStarted","Data":"753e99a9bcbbb90c45d473d1de240f41b2b66e0f1ca51f090dfd4d6abcd2ab3c"}
Jul 18 00:41:47.604953 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.604565    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:47.607272 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.606888    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k28lr" event={"ID":"9d36b8cf-0af7-46ad-a1d3-b4456fbba020","Type":"ContainerStarted","Data":"dffe6ac400433dd79588f22eaf7c9ce337670f96ea2de2b9d34b40b5fe4428d1"}
Jul 18 00:41:47.612333 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.612265    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tccd" event={"ID":"55e9d85e-ba32-4a40-b6d5-b9931ecbd044","Type":"ContainerStarted","Data":"0e8821f96b1919e5fc105d6bae80bacfb2031c59311463260d217c0ed2bf4842"}
Jul 18 00:41:47.623112 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.623047    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kw5b5" podStartSLOduration=1.105987061 podStartE2EDuration="6.62302429s" totalImagesPullingTime="5.517037229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:47.622697231 +0000 UTC m=+63.748632211" watchObservedRunningTime="2026-07-18 00:41:47.62302429 +0000 UTC m=+63.748959274"
Jul 18 00:41:47.649933 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.649836    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k28lr" podStartSLOduration=1.036991518 podStartE2EDuration="6.649813385s" totalImagesPullingTime="5.612821867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:47.649322228 +0000 UTC m=+63.775257206" watchObservedRunningTime="2026-07-18 00:41:47.649813385 +0000 UTC m=+63.775748366"
Jul 18 00:41:47.688427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.688395    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"]
Jul 18 00:41:47.691899 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:47.691872    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069929f9_0ece_4af0_a944_5cba5faaf972.slice/crio-406e077a9f298ea8f358bc8bd843c18cd8832b918b773d8a3517d886a97dbbff WatchSource:0}: Error finding container 406e077a9f298ea8f358bc8bd843c18cd8832b918b773d8a3517d886a97dbbff: Status 404 returned error can't find the container with id 406e077a9f298ea8f358bc8bd843c18cd8832b918b773d8a3517d886a97dbbff
Jul 18 00:41:47.697697 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.697676    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-serving-cert\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.697799 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.697702    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-service-ca\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.697799 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.697739    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-oauth-config\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.697799 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.697786    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-config\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.697972 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.697809    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-oauth-serving-cert\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.697972 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.697895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvk68\" (UniqueName: \"kubernetes.io/projected/bc6be778-f6b7-485e-8e76-021a4dc37e95-kube-api-access-jvk68\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.798891 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.798861    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvk68\" (UniqueName: \"kubernetes.io/projected/bc6be778-f6b7-485e-8e76-021a4dc37e95-kube-api-access-jvk68\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.799049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.798912    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-serving-cert\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.799049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.798936    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-service-ca\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.799049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.798966    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-oauth-config\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.799049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.798991    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-config\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.799049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.799020    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-oauth-serving-cert\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.799908 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.799875    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-config\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.800031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.799887    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-service-ca\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.800130 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.800108    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-oauth-serving-cert\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.801581 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.801559    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-oauth-config\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.801679 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.801586    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-serving-cert\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.806848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.806797    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvk68\" (UniqueName: \"kubernetes.io/projected/bc6be778-f6b7-485e-8e76-021a4dc37e95-kube-api-access-jvk68\") pod \"console-5c4fdd577b-f4x9g\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") " pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:47.865791 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:47.865751    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:48.027877 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:48.025740    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4fdd577b-f4x9g"]
Jul 18 00:41:48.030563 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:48.030534    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6be778_f6b7_485e_8e76_021a4dc37e95.slice/crio-7d5c9f7329e61a17bf26f3a7f108333a6cc865535ec89958f704c015939bb43e WatchSource:0}: Error finding container 7d5c9f7329e61a17bf26f3a7f108333a6cc865535ec89958f704c015939bb43e: Status 404 returned error can't find the container with id 7d5c9f7329e61a17bf26f3a7f108333a6cc865535ec89958f704c015939bb43e
Jul 18 00:41:48.617036 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:48.616960    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd" event={"ID":"069929f9-0ece-4af0-a944-5cba5faaf972","Type":"ContainerStarted","Data":"406e077a9f298ea8f358bc8bd843c18cd8832b918b773d8a3517d886a97dbbff"}
Jul 18 00:41:48.618790 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:48.618719    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fdd577b-f4x9g" event={"ID":"bc6be778-f6b7-485e-8e76-021a4dc37e95","Type":"ContainerStarted","Data":"7d5c9f7329e61a17bf26f3a7f108333a6cc865535ec89958f704c015939bb43e"}
Jul 18 00:41:49.624335 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:49.624287    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd" event={"ID":"069929f9-0ece-4af0-a944-5cba5faaf972","Type":"ContainerStarted","Data":"740821a7fc0d4f4f95d8adf5e656edef6d59604af3f14d022c85247904bce294"}
Jul 18 00:41:49.624813 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:49.624673    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:49.630160 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:49.630133    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd"
Jul 18 00:41:49.653821 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:49.653758    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v5swd" podStartSLOduration=2.505329313 podStartE2EDuration="3.65374153s" totalImagesPullingTime="1.148412217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:49.652730412 +0000 UTC m=+65.778665391" watchObservedRunningTime="2026-07-18 00:41:49.65374153 +0000 UTC m=+65.779676487"
Jul 18 00:41:50.020260 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.020220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:50.023064 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.023036    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 18 00:41:50.033779 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.033747    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a089e7e1-1597-4899-83b2-357a35bb40cb-original-pull-secret\") pod \"global-pull-secret-syncer-tc4gd\" (UID: \"a089e7e1-1597-4899-83b2-357a35bb40cb\") " pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:50.120781 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.120745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:50.123547 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.123523    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 18 00:41:50.134558 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.134525    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd69c75-0071-4835-a044-9accda5ecee4-metrics-certs\") pod \"network-metrics-daemon-4p62b\" (UID: \"2bd69c75-0071-4835-a044-9accda5ecee4\") " pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:50.149354 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.149319    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc4gd"
Jul 18 00:41:50.168295 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.168266    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9tdp4\""
Jul 18 00:41:50.176694 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.176658    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4p62b"
Jul 18 00:41:50.221235 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.221201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:50.223831 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.223810    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 18 00:41:50.234018 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.233990    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 18 00:41:50.244400 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.244376    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv55t\" (UniqueName: \"kubernetes.io/projected/f5918960-9d76-46ac-8ce5-82c432a9e284-kube-api-access-pv55t\") pod \"network-check-target-qqqj9\" (UID: \"f5918960-9d76-46ac-8ce5-82c432a9e284\") " pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:50.460142 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.460111    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2stqj\""
Jul 18 00:41:50.468190 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.468166    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:50.708805 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.708722    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"]
Jul 18 00:41:50.728320 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.728286    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.732786 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.732748    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 18 00:41:50.733919 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.733895    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 18 00:41:50.734033 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.733947    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 18 00:41:50.734113 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.733897    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g7f8k\""
Jul 18 00:41:50.734193 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.734136    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 18 00:41:50.734193 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.734165    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 18 00:41:50.735866 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.735802    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"]
Jul 18 00:41:50.826384 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.826348    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.826551 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.826395    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvxg\" (UniqueName: \"kubernetes.io/projected/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-kube-api-access-hxvxg\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.826551 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.826456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.826551 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.826520    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.927668 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.927611    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.927858 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.927745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.927858 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.927785    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.927858 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:50.927795    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 18 00:41:50.927858 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.927812    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvxg\" (UniqueName: \"kubernetes.io/projected/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-kube-api-access-hxvxg\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.928152 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:50.927875    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-tls podName:2dea8cd9-8a03-4db8-9d07-230fcb4ca17e nodeName:}" failed. No retries permitted until 2026-07-18 00:41:51.427852597 +0000 UTC m=+67.553787575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-btpwm" (UID: "2dea8cd9-8a03-4db8-9d07-230fcb4ca17e") : secret "prometheus-operator-tls" not found
Jul 18 00:41:50.928600 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.928577    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.930406 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.930382    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:50.936955 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:50.936925    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvxg\" (UniqueName: \"kubernetes.io/projected/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-kube-api-access-hxvxg\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:51.432586 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:51.432545    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:51.435290 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:51.435262    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2dea8cd9-8a03-4db8-9d07-230fcb4ca17e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-btpwm\" (UID: \"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:51.639029 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:51.638988    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"
Jul 18 00:41:52.351547 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.351349    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qqqj9"]
Jul 18 00:41:52.354581 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:52.354548    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5918960_9d76_46ac_8ce5_82c432a9e284.slice/crio-217872326aff0104fa2544258a290938448b4c926299194d4b62d8065914a8e1 WatchSource:0}: Error finding container 217872326aff0104fa2544258a290938448b4c926299194d4b62d8065914a8e1: Status 404 returned error can't find the container with id 217872326aff0104fa2544258a290938448b4c926299194d4b62d8065914a8e1
Jul 18 00:41:52.576183 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.576149    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-btpwm"]
Jul 18 00:41:52.579108 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.579073    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tc4gd"]
Jul 18 00:41:52.579996 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.579971    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4p62b"]
Jul 18 00:41:52.581066 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:52.581044    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dea8cd9_8a03_4db8_9d07_230fcb4ca17e.slice/crio-ed16c364b5908c7be12a951837b0ce1af3fa44e4032b9bc37ad0698ac7f31376 WatchSource:0}: Error finding container ed16c364b5908c7be12a951837b0ce1af3fa44e4032b9bc37ad0698ac7f31376: Status 404 returned error can't find the container with id ed16c364b5908c7be12a951837b0ce1af3fa44e4032b9bc37ad0698ac7f31376
Jul 18 00:41:52.583437 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:52.583410    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda089e7e1_1597_4899_83b2_357a35bb40cb.slice/crio-628297d31a7d8c4b83d947535f6d653308768ca90849b7ad2efd382dbf32208a WatchSource:0}: Error finding container 628297d31a7d8c4b83d947535f6d653308768ca90849b7ad2efd382dbf32208a: Status 404 returned error can't find the container with id 628297d31a7d8c4b83d947535f6d653308768ca90849b7ad2efd382dbf32208a
Jul 18 00:41:52.584261 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:52.584235    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd69c75_0071_4835_a044_9accda5ecee4.slice/crio-22bc426728d536a144d0deb1a121f920e8f2020827f3b0b929c86561d81ebf71 WatchSource:0}: Error finding container 22bc426728d536a144d0deb1a121f920e8f2020827f3b0b929c86561d81ebf71: Status 404 returned error can't find the container with id 22bc426728d536a144d0deb1a121f920e8f2020827f3b0b929c86561d81ebf71
Jul 18 00:41:52.634369 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.634295    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm" event={"ID":"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e","Type":"ContainerStarted","Data":"ed16c364b5908c7be12a951837b0ce1af3fa44e4032b9bc37ad0698ac7f31376"}
Jul 18 00:41:52.635461 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.635430    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tc4gd" event={"ID":"a089e7e1-1597-4899-83b2-357a35bb40cb","Type":"ContainerStarted","Data":"628297d31a7d8c4b83d947535f6d653308768ca90849b7ad2efd382dbf32208a"}
Jul 18 00:41:52.637144 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.637119    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tccd" event={"ID":"55e9d85e-ba32-4a40-b6d5-b9931ecbd044","Type":"ContainerStarted","Data":"f8afc62b9cfad3567a6f16aeed7ef38651b1512322d0511064a59c3a2ab445fa"}
Jul 18 00:41:52.638482 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.638464    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fdd577b-f4x9g" event={"ID":"bc6be778-f6b7-485e-8e76-021a4dc37e95","Type":"ContainerStarted","Data":"e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055"}
Jul 18 00:41:52.639562 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.639540    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4p62b" event={"ID":"2bd69c75-0071-4835-a044-9accda5ecee4","Type":"ContainerStarted","Data":"22bc426728d536a144d0deb1a121f920e8f2020827f3b0b929c86561d81ebf71"}
Jul 18 00:41:52.640541 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.640521    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qqqj9" event={"ID":"f5918960-9d76-46ac-8ce5-82c432a9e284","Type":"ContainerStarted","Data":"217872326aff0104fa2544258a290938448b4c926299194d4b62d8065914a8e1"}
Jul 18 00:41:52.655891 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:52.655845    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c4fdd577b-f4x9g" podStartSLOduration=1.509314844 podStartE2EDuration="5.655832403s" totalImagesPullingTime="4.146517559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:52.65532205 +0000 UTC m=+68.781257030" watchObservedRunningTime="2026-07-18 00:41:52.655832403 +0000 UTC m=+68.781767382"
Jul 18 00:41:54.656644 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:54.655892    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm" event={"ID":"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e","Type":"ContainerStarted","Data":"62fbf94e6e00555b21921a7598a6ce4df4f823ab2cdc5e829a04f0bf757075aa"}
Jul 18 00:41:54.656644 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:54.656009    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm" event={"ID":"2dea8cd9-8a03-4db8-9d07-230fcb4ca17e","Type":"ContainerStarted","Data":"0ff640b54ab3b6f4df3a0261323a058e684ed4ed78981a4576fce6fa5c1d9359"}
Jul 18 00:41:54.663679 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:54.663607    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4p62b" event={"ID":"2bd69c75-0071-4835-a044-9accda5ecee4","Type":"ContainerStarted","Data":"812b6ee5efbeb2921d1319d4e33888c784f38699a3e3afc7ca8fa107f0d3f25e"}
Jul 18 00:41:54.663849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:54.663685    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4p62b" event={"ID":"2bd69c75-0071-4835-a044-9accda5ecee4","Type":"ContainerStarted","Data":"3d2bafbc597ad6463b0625513f2564bd01fa374f8528cc78ae826ae7cf44cc05"}
Jul 18 00:41:54.678853 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:54.678377    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-btpwm" podStartSLOduration=2.9464128499999998 podStartE2EDuration="4.678355799s" totalImagesPullingTime="1.731942949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:54.676794294 +0000 UTC m=+70.802729275" watchObservedRunningTime="2026-07-18 00:41:54.678355799 +0000 UTC m=+70.804290783"
Jul 18 00:41:54.692138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:54.691740    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4p62b" podStartSLOduration=68.966435985 podStartE2EDuration="1m10.691710011s" totalImagesPullingTime="1.725274026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:54.690924307 +0000 UTC m=+70.816859287" watchObservedRunningTime="2026-07-18 00:41:54.691710011 +0000 UTC m=+70.817645027"
Jul 18 00:41:55.329106 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.329070    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55547d65ff-hc2st"]
Jul 18 00:41:55.333844 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.333820    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.342417 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.342357    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 18 00:41:55.343528 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.343500    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55547d65ff-hc2st"]
Jul 18 00:41:55.464273 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.464189    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ksf\" (UniqueName: \"kubernetes.io/projected/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-kube-api-access-q5ksf\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.464273 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.464234    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-oauth-config\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.464273 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.464263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-service-ca\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.464524 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.464292    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-config\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.464524 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.464316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-trusted-ca-bundle\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.464524 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.464336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-oauth-serving-cert\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.464524 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.464454    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-serving-cert\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.565461 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.565425    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-oauth-serving-cert\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.565657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.565505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-serving-cert\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.565657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.565561    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ksf\" (UniqueName: \"kubernetes.io/projected/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-kube-api-access-q5ksf\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.565657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.565588    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-oauth-config\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.565657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.565635    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-service-ca\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.565870 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.565662    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-config\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.565870 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.565688    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-trusted-ca-bundle\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.566239 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.566211    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-oauth-serving-cert\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.566499 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.566476    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-trusted-ca-bundle\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.567224 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.566592    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-service-ca\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.567460 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.567190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-config\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.568921 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.568891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-oauth-config\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.569417 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.569397    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-serving-cert\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.574000 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.573955    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ksf\" (UniqueName: \"kubernetes.io/projected/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-kube-api-access-q5ksf\") pod \"console-55547d65ff-hc2st\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") " pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:55.647256 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:55.647186    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:41:57.054091 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.054054    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"]
Jul 18 00:41:57.058556 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.058523    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.064609 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.064587    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dqj92\""
Jul 18 00:41:57.065293 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.065274    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 18 00:41:57.065873 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.065854    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 18 00:41:57.071044 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.071021    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"]
Jul 18 00:41:57.075647 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.075607    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8mtpf"]
Jul 18 00:41:57.081205 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.081181    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.084634 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.084598    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 18 00:41:57.084723 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.084676    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 18 00:41:57.084790 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.084755    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wdls5\""
Jul 18 00:41:57.085459 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.085432    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 18 00:41:57.095378 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.095353    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-x744k"]
Jul 18 00:41:57.099850 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.099831    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.102737 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.102717    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5bgff\""
Jul 18 00:41:57.108593 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.108548    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 18 00:41:57.108593 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.108586    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 18 00:41:57.108762 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.108551    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 18 00:41:57.120349 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.120322    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-x744k"]
Jul 18 00:41:57.177077 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177042    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-textfile\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.177262 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177087    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.177262 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-sys\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.177262 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e284174e-dce4-4f66-8f8c-c58ae7038721-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.177262 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177188    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e282b8c-73e9-491f-bfa5-46be537dd4d4-metrics-client-ca\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.177262 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177232    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.177559 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177262    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.177559 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177314    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.177559 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177337    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.177559 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177404    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.177559 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177490    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.177823 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-root\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.177823 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177632    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5p2q\" (UniqueName: \"kubernetes.io/projected/e284174e-dce4-4f66-8f8c-c58ae7038721-kube-api-access-n5p2q\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.177973 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.178100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.177981    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cqq\" (UniqueName: \"kubernetes.io/projected/6e282b8c-73e9-491f-bfa5-46be537dd4d4-kube-api-access-z4cqq\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.178100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.178009    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c77e01c-9265-452d-b94d-30c8eb6a6b56-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.178100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.178096    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c77e01c-9265-452d-b94d-30c8eb6a6b56-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.178251 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.178123    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-wtmp\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.178251 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.178147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6rc\" (UniqueName: \"kubernetes.io/projected/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-api-access-rx6rc\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.278570 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278530    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.278770 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278586    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.278770 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-root\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.278770 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278672    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5p2q\" (UniqueName: \"kubernetes.io/projected/e284174e-dce4-4f66-8f8c-c58ae7038721-kube-api-access-n5p2q\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.278770 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.278770 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278718    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cqq\" (UniqueName: \"kubernetes.io/projected/6e282b8c-73e9-491f-bfa5-46be537dd4d4-kube-api-access-z4cqq\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.278770 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c77e01c-9265-452d-b94d-30c8eb6a6b56-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.278770 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278743    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-root\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278810    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c77e01c-9265-452d-b94d-30c8eb6a6b56-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278844    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-wtmp\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278868    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6rc\" (UniqueName: \"kubernetes.io/projected/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-api-access-rx6rc\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-textfile\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278930    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.278944    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278962    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-sys\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.278987    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e284174e-dce4-4f66-8f8c-c58ae7038721-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.279010    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls podName:6e282b8c-73e9-491f-bfa5-46be537dd4d4 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:57.778987582 +0000 UTC m=+73.904922542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls") pod "node-exporter-8mtpf" (UID: "6e282b8c-73e9-491f-bfa5-46be537dd4d4") : secret "node-exporter-tls" not found
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e282b8c-73e9-491f-bfa5-46be537dd4d4-metrics-client-ca\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279069    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279102 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279097    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.279729 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279133    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.279729 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279158    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279729 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279500    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-wtmp\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.279729 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c77e01c-9265-452d-b94d-30c8eb6a6b56-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.279729 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e284174e-dce4-4f66-8f8c-c58ae7038721-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.279729 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279720    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e282b8c-73e9-491f-bfa5-46be537dd4d4-sys\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.280025 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.279970    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-textfile\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.280083 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.280057    2554 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 18 00:41:57.280137 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.280094    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.280137 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.280106    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-tls podName:8c77e01c-9265-452d-b94d-30c8eb6a6b56 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:57.780090628 +0000 UTC m=+73.906025601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-x744k" (UID: "8c77e01c-9265-452d-b94d-30c8eb6a6b56") : secret "kube-state-metrics-tls" not found
Jul 18 00:41:57.280253 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.280171    2554 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 18 00:41:57.280253 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.280173    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e282b8c-73e9-491f-bfa5-46be537dd4d4-metrics-client-ca\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.280253 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.280216    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-tls podName:e284174e-dce4-4f66-8f8c-c58ae7038721 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:57.780206436 +0000 UTC m=+73.906141393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-tllg7" (UID: "e284174e-dce4-4f66-8f8c-c58ae7038721") : secret "openshift-state-metrics-tls" not found
Jul 18 00:41:57.280253 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.280209    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.280792 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.280741    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c77e01c-9265-452d-b94d-30c8eb6a6b56-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.281415 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.281396    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.282435 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.282393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.283239 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.283214    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.287786 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.287763    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5p2q\" (UniqueName: \"kubernetes.io/projected/e284174e-dce4-4f66-8f8c-c58ae7038721-kube-api-access-n5p2q\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.287981 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.287957    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cqq\" (UniqueName: \"kubernetes.io/projected/6e282b8c-73e9-491f-bfa5-46be537dd4d4-kube-api-access-z4cqq\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.288274 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.288252    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6rc\" (UniqueName: \"kubernetes.io/projected/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-api-access-rx6rc\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.621569 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.621535    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kw5b5"
Jul 18 00:41:57.784432 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.784395    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:57.784596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.784499    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.784596 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.784524    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 18 00:41:57.784596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.784535    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.784596 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:41:57.784595    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls podName:6e282b8c-73e9-491f-bfa5-46be537dd4d4 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:58.78457385 +0000 UTC m=+74.910508813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls") pod "node-exporter-8mtpf" (UID: "6e282b8c-73e9-491f-bfa5-46be537dd4d4") : secret "node-exporter-tls" not found
Jul 18 00:41:57.787263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.787236    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c77e01c-9265-452d-b94d-30c8eb6a6b56-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-x744k\" (UID: \"8c77e01c-9265-452d-b94d-30c8eb6a6b56\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:57.787461 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.787440    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e284174e-dce4-4f66-8f8c-c58ae7038721-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-tllg7\" (UID: \"e284174e-dce4-4f66-8f8c-c58ae7038721\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:57.867077 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.867037    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:57.867282 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.867109    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:57.872588 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.872515    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:57.970753 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:57.970707    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"
Jul 18 00:41:58.009327 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.009289    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k"
Jul 18 00:41:58.150460 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.150355    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:41:58.156072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.155367    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.158380    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.158502    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.159430    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jts4m\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.159498    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.159532    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.159495    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.159887    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.159991    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.160177    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 18 00:41:58.160993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.160399    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 18 00:41:58.170990 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.170965    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:41:58.289433 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289399    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chqg\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-kube-api-access-2chqg\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289637 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289460    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289637 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289500    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-config-out\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289637 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289531    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289637 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289562    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289637 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289591    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289909 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289705    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289909 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289750    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289909 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289815    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289909 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289860    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.289909 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.290318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289934    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.290318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.289965    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-web-config\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390433 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390393    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390436    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390494    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390520    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390569    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-web-config\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390653    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chqg\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-kube-api-access-2chqg\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390701    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390733    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-config-out\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390757    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390785    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390812    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.390896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.390846    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.391773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.391323    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.391773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.391446    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.391949 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.391841    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.394224 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.393599    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.394224 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.393701    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.394224 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.394166    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.394224 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.394184    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-web-config\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.394795 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.394771    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-config-out\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.395026 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.395006    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.395131 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.395033    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.395476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.395428    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.395576 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.395524    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.399580 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.399554    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chqg\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-kube-api-access-2chqg\") pod \"alertmanager-main-0\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.469487 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.469405    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:41:58.687460 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.687429    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:41:58.794968 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.794827    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:58.797285 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.797262    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e282b8c-73e9-491f-bfa5-46be537dd4d4-node-exporter-tls\") pod \"node-exporter-8mtpf\" (UID: \"6e282b8c-73e9-491f-bfa5-46be537dd4d4\") " pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:58.892096 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:58.892065    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mtpf"
Jul 18 00:41:59.066698 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.066581    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b987bb95b-mctr7"]
Jul 18 00:41:59.070103 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.070079    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.072803 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.072747    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 18 00:41:59.072924 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.072839    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 18 00:41:59.072924 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.072870    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 18 00:41:59.073008 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.072982    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5en8nkuqv0fck\""
Jul 18 00:41:59.073008 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.072986    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-f7xjj\""
Jul 18 00:41:59.073095 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.073068    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 18 00:41:59.073184 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.073168    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 18 00:41:59.084039 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.084012    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b987bb95b-mctr7"]
Jul 18 00:41:59.199082 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199048    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.199521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199099    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.199521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199124    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-tls\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.199521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199146    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2rkw\" (UniqueName: \"kubernetes.io/projected/17809a4e-df1b-40b8-9b2f-10f489d32b97-kube-api-access-s2rkw\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.199521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-grpc-tls\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.199521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199283    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.199521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17809a4e-df1b-40b8-9b2f-10f489d32b97-metrics-client-ca\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.199521 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.199355    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.301946    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17809a4e-df1b-40b8-9b2f-10f489d32b97-metrics-client-ca\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.301998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.302046    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.302107    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.302145    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-tls\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.302177    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2rkw\" (UniqueName: \"kubernetes.io/projected/17809a4e-df1b-40b8-9b2f-10f489d32b97-kube-api-access-s2rkw\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.302224    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-grpc-tls\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.303739 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.302261    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.304810 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.304748    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17809a4e-df1b-40b8-9b2f-10f489d32b97-metrics-client-ca\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.323532 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.319582    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.323532 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.320096    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.323532 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.322092    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2rkw\" (UniqueName: \"kubernetes.io/projected/17809a4e-df1b-40b8-9b2f-10f489d32b97-kube-api-access-s2rkw\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.332842 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.332465    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.334044 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.333965    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.340958 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.335563    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-grpc-tls\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.344092 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.344037    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17809a4e-df1b-40b8-9b2f-10f489d32b97-secret-thanos-querier-tls\") pod \"thanos-querier-6b987bb95b-mctr7\" (UID: \"17809a4e-df1b-40b8-9b2f-10f489d32b97\") " pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.381369 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.381333    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:41:59.437297 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.437268    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-x744k"]
Jul 18 00:41:59.562971 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.561598    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b987bb95b-mctr7"]
Jul 18 00:41:59.564604 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:59.564560    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17809a4e_df1b_40b8_9b2f_10f489d32b97.slice/crio-176d1450c6eba291375ef8edf244c8d9f2536ed11716c72298c8b6dd6ffc2479 WatchSource:0}: Error finding container 176d1450c6eba291375ef8edf244c8d9f2536ed11716c72298c8b6dd6ffc2479: Status 404 returned error can't find the container with id 176d1450c6eba291375ef8edf244c8d9f2536ed11716c72298c8b6dd6ffc2479
Jul 18 00:41:59.645880 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.645712    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7"]
Jul 18 00:41:59.649245 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:59.649214    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode284174e_dce4_4f66_8f8c_c58ae7038721.slice/crio-baabcb7758e3aab8e74727fb4a9b841fdf77986fbbdefbc7312700565e14026e WatchSource:0}: Error finding container baabcb7758e3aab8e74727fb4a9b841fdf77986fbbdefbc7312700565e14026e: Status 404 returned error can't find the container with id baabcb7758e3aab8e74727fb4a9b841fdf77986fbbdefbc7312700565e14026e
Jul 18 00:41:59.652658 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.652627    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55547d65ff-hc2st"]
Jul 18 00:41:59.656020 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:59.655992    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd338c2f8_0ced_4a1a_9caf_caf99d335d0b.slice/crio-bc82be9b2a1651275ba0a180812d7dcd34263deda2378b7f094c99c012c0eb60 WatchSource:0}: Error finding container bc82be9b2a1651275ba0a180812d7dcd34263deda2378b7f094c99c012c0eb60: Status 404 returned error can't find the container with id bc82be9b2a1651275ba0a180812d7dcd34263deda2378b7f094c99c012c0eb60
Jul 18 00:41:59.666773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.666672    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:41:59.669787 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:41:59.669757    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922e1204_62c1_4e57_9a71_7b9f259139a9.slice/crio-139742fff601f09e72f00db61721eacb966e073fda21e2aeacedf2a9b21a489c WatchSource:0}: Error finding container 139742fff601f09e72f00db61721eacb966e073fda21e2aeacedf2a9b21a489c: Status 404 returned error can't find the container with id 139742fff601f09e72f00db61721eacb966e073fda21e2aeacedf2a9b21a489c
Jul 18 00:41:59.688218 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.688146    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tc4gd" event={"ID":"a089e7e1-1597-4899-83b2-357a35bb40cb","Type":"ContainerStarted","Data":"42f1367e7024866932e358e0a1e4863594d7736ec503613b18fbf851db65170f"}
Jul 18 00:41:59.691592 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.691539    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tccd" event={"ID":"55e9d85e-ba32-4a40-b6d5-b9931ecbd044","Type":"ContainerStarted","Data":"e9b93c79e211b700406d7f8a66b29d770a697ca6ded40f996faaf7d4eb40b4aa"}
Jul 18 00:41:59.692950 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.692927    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" event={"ID":"17809a4e-df1b-40b8-9b2f-10f489d32b97","Type":"ContainerStarted","Data":"176d1450c6eba291375ef8edf244c8d9f2536ed11716c72298c8b6dd6ffc2479"}
Jul 18 00:41:59.694184 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.694165    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7" event={"ID":"e284174e-dce4-4f66-8f8c-c58ae7038721","Type":"ContainerStarted","Data":"baabcb7758e3aab8e74727fb4a9b841fdf77986fbbdefbc7312700565e14026e"}
Jul 18 00:41:59.695657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.695608    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerStarted","Data":"139742fff601f09e72f00db61721eacb966e073fda21e2aeacedf2a9b21a489c"}
Jul 18 00:41:59.696870 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.696841    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mtpf" event={"ID":"6e282b8c-73e9-491f-bfa5-46be537dd4d4","Type":"ContainerStarted","Data":"85b3b8dfb55a91cf5cd8773b3adcf72fcaa48c57ebe08d8b05c4c20c571650c1"}
Jul 18 00:41:59.698263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.698238    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55547d65ff-hc2st" event={"ID":"d338c2f8-0ced-4a1a-9caf-caf99d335d0b","Type":"ContainerStarted","Data":"bc82be9b2a1651275ba0a180812d7dcd34263deda2378b7f094c99c012c0eb60"}
Jul 18 00:41:59.699477 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.699456    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k" event={"ID":"8c77e01c-9265-452d-b94d-30c8eb6a6b56","Type":"ContainerStarted","Data":"6fd709429e707ffcf2458c5cd5ecc0d0fc5bcacc83a3f980647dd20c8e3eeac1"}
Jul 18 00:41:59.703066 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.702776    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qqqj9" event={"ID":"f5918960-9d76-46ac-8ce5-82c432a9e284","Type":"ContainerStarted","Data":"4ee5b0c2468a4ebadc780578fa842270fb5ec2639590d596ff0b97d16a077db0"}
Jul 18 00:41:59.703066 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.703000    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:41:59.705130 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.705078    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tc4gd" podStartSLOduration=69.05121469 podStartE2EDuration="1m15.705059803s" totalImagesPullingTime="6.653845113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:59.70374516 +0000 UTC m=+75.829680141" watchObservedRunningTime="2026-07-18 00:41:59.705059803 +0000 UTC m=+75.830994784"
Jul 18 00:41:59.719346 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:41:59.719291    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qqqj9" podStartSLOduration=68.841095071 podStartE2EDuration="1m15.719273117s" totalImagesPullingTime="6.878178046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:59.719204784 +0000 UTC m=+75.845139764" watchObservedRunningTime="2026-07-18 00:41:59.719273117 +0000 UTC m=+75.845208095"
Jul 18 00:42:00.710290 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:00.710251    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7" event={"ID":"e284174e-dce4-4f66-8f8c-c58ae7038721","Type":"ContainerStarted","Data":"01fbcdb7190d750bbaa623efe541d34b20992cc6e9e5ff78915879864fd67221"}
Jul 18 00:42:00.710907 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:00.710316    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7" event={"ID":"e284174e-dce4-4f66-8f8c-c58ae7038721","Type":"ContainerStarted","Data":"8c210df25c32cdd19d4149f84ca9dec3d7ca769ac7d6d2e4990db52e11944b50"}
Jul 18 00:42:00.712583 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:00.712549    2554 generic.go:356] "Generic (PLEG): container finished" podID="6e282b8c-73e9-491f-bfa5-46be537dd4d4" containerID="854ca19ae7733b30f374ff9c7cc607928cefe326b3f2ebf1bd704293cd3b2803" exitCode=0
Jul 18 00:42:00.712710 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:00.712652    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mtpf" event={"ID":"6e282b8c-73e9-491f-bfa5-46be537dd4d4","Type":"ContainerDied","Data":"854ca19ae7733b30f374ff9c7cc607928cefe326b3f2ebf1bd704293cd3b2803"}
Jul 18 00:42:00.716223 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:00.716196    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55547d65ff-hc2st" event={"ID":"d338c2f8-0ced-4a1a-9caf-caf99d335d0b","Type":"ContainerStarted","Data":"c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8"}
Jul 18 00:42:00.736777 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:00.736602    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6tccd" podStartSLOduration=2.3957222209999998 podStartE2EDuration="19.736585236s" totalImagesPullingTime="17.340863015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:41:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:59.75044116 +0000 UTC m=+75.876376143" watchObservedRunningTime="2026-07-18 00:42:00.736585236 +0000 UTC m=+76.862520215"
Jul 18 00:42:00.750497 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:00.750370    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55547d65ff-hc2st" podStartSLOduration=5.7503541259999995 podStartE2EDuration="5.750354126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:00.750029688 +0000 UTC m=+76.875964668" watchObservedRunningTime="2026-07-18 00:42:00.750354126 +0000 UTC m=+76.876289106"
Jul 18 00:42:01.484926 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.484892    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"]
Jul 18 00:42:01.488277 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.488254    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.491107 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.491076    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-knqjk\""
Jul 18 00:42:01.491276 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.491182    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 18 00:42:01.492034 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.492008    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 18 00:42:01.492153 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.492081    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 18 00:42:01.492211 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.492162    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 18 00:42:01.492266 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.492244    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f382fufk64aht\""
Jul 18 00:42:01.499516 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.499493    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"]
Jul 18 00:42:01.625649 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.625596    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-secret-metrics-server-tls\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.625812 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.625661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a2040fa1-7916-4610-bfcb-445d2faa646e-metrics-server-audit-profiles\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.625812 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.625717    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-client-ca-bundle\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.625812 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.625751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbv4\" (UniqueName: \"kubernetes.io/projected/a2040fa1-7916-4610-bfcb-445d2faa646e-kube-api-access-2vbv4\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.625978 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.625848    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a2040fa1-7916-4610-bfcb-445d2faa646e-audit-log\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.625978 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.625900    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-secret-metrics-server-client-certs\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.625978 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.625968    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2040fa1-7916-4610-bfcb-445d2faa646e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.726948 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.726917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a2040fa1-7916-4610-bfcb-445d2faa646e-audit-log\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.726959    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-secret-metrics-server-client-certs\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.726992    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2040fa1-7916-4610-bfcb-445d2faa646e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.727042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-secret-metrics-server-tls\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.727058    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a2040fa1-7916-4610-bfcb-445d2faa646e-metrics-server-audit-profiles\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.727100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-client-ca-bundle\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.727246    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbv4\" (UniqueName: \"kubernetes.io/projected/a2040fa1-7916-4610-bfcb-445d2faa646e-kube-api-access-2vbv4\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.727387    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a2040fa1-7916-4610-bfcb-445d2faa646e-audit-log\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.727940 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.727876    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2040fa1-7916-4610-bfcb-445d2faa646e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.728200 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.728176    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a2040fa1-7916-4610-bfcb-445d2faa646e-metrics-server-audit-profiles\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.730155 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.730132    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-secret-metrics-server-client-certs\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.730247 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.730183    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-client-ca-bundle\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.730327 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.730307    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a2040fa1-7916-4610-bfcb-445d2faa646e-secret-metrics-server-tls\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.735859 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.735804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbv4\" (UniqueName: \"kubernetes.io/projected/a2040fa1-7916-4610-bfcb-445d2faa646e-kube-api-access-2vbv4\") pod \"metrics-server-7c5ccd67dc-wk5wk\" (UID: \"a2040fa1-7916-4610-bfcb-445d2faa646e\") " pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.801572 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.801534    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:01.906580 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.906510    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"]
Jul 18 00:42:01.910468 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.910443    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:01.912989 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.912969    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g2jwt\""
Jul 18 00:42:01.913923 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.913904    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 18 00:42:01.920180 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:01.920116    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"]
Jul 18 00:42:02.031675 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.031639    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkq2t\" (UniqueName: \"kubernetes.io/projected/f7e8eece-45ff-4481-b41a-bad92ad7fdad-kube-api-access-vkq2t\") pod \"monitoring-plugin-7b7b7bc8bf-8wj2j\" (UID: \"f7e8eece-45ff-4481-b41a-bad92ad7fdad\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.031860 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.031746    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7e8eece-45ff-4481-b41a-bad92ad7fdad-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8wj2j\" (UID: \"f7e8eece-45ff-4481-b41a-bad92ad7fdad\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.132415 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.132387    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7e8eece-45ff-4481-b41a-bad92ad7fdad-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8wj2j\" (UID: \"f7e8eece-45ff-4481-b41a-bad92ad7fdad\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.132536 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.132452    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkq2t\" (UniqueName: \"kubernetes.io/projected/f7e8eece-45ff-4481-b41a-bad92ad7fdad-kube-api-access-vkq2t\") pod \"monitoring-plugin-7b7b7bc8bf-8wj2j\" (UID: \"f7e8eece-45ff-4481-b41a-bad92ad7fdad\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.132579 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:42:02.132544    2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 18 00:42:02.132610 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:42:02.132601    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e8eece-45ff-4481-b41a-bad92ad7fdad-monitoring-plugin-cert podName:f7e8eece-45ff-4481-b41a-bad92ad7fdad nodeName:}" failed. No retries permitted until 2026-07-18 00:42:02.632584719 +0000 UTC m=+78.758519679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f7e8eece-45ff-4481-b41a-bad92ad7fdad-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-8wj2j" (UID: "f7e8eece-45ff-4481-b41a-bad92ad7fdad") : secret "monitoring-plugin-cert" not found
Jul 18 00:42:02.149109 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.146909    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkq2t\" (UniqueName: \"kubernetes.io/projected/f7e8eece-45ff-4481-b41a-bad92ad7fdad-kube-api-access-vkq2t\") pod \"monitoring-plugin-7b7b7bc8bf-8wj2j\" (UID: \"f7e8eece-45ff-4481-b41a-bad92ad7fdad\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.248041 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.247033    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"]
Jul 18 00:42:02.637483 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.637382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7e8eece-45ff-4481-b41a-bad92ad7fdad-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8wj2j\" (UID: \"f7e8eece-45ff-4481-b41a-bad92ad7fdad\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.639745 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.639718    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7e8eece-45ff-4481-b41a-bad92ad7fdad-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-8wj2j\" (UID: \"f7e8eece-45ff-4481-b41a-bad92ad7fdad\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.730331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.730288    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk" event={"ID":"a2040fa1-7916-4610-bfcb-445d2faa646e","Type":"ContainerStarted","Data":"53b047afca17701d54a001bee67e5c2007bd264b30a186edcc5ba6ee3767a6fa"}
Jul 18 00:42:02.732316 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.732290    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" event={"ID":"17809a4e-df1b-40b8-9b2f-10f489d32b97","Type":"ContainerStarted","Data":"abac7179172689c298a64cf2a1ca4028c70943baadc852fdcf8f7ce69cccf346"}
Jul 18 00:42:02.732438 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.732345    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" event={"ID":"17809a4e-df1b-40b8-9b2f-10f489d32b97","Type":"ContainerStarted","Data":"bc0b3bd33a38826f828534f727235a1109f9b696ca7352d02395d319c41fb115"}
Jul 18 00:42:02.732483 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.732444    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" event={"ID":"17809a4e-df1b-40b8-9b2f-10f489d32b97","Type":"ContainerStarted","Data":"35b64997da15b2f43bdd1d5d907732f9f5fc5e33a02ed266a0c88ac5491fd79c"}
Jul 18 00:42:02.734265 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.734241    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7" event={"ID":"e284174e-dce4-4f66-8f8c-c58ae7038721","Type":"ContainerStarted","Data":"a1914a1994a4ac4b2c61a04f2961e2271da41c42206b48df83a0748fe2501263"}
Jul 18 00:42:02.735707 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.735687    2554 generic.go:356] "Generic (PLEG): container finished" podID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerID="c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890" exitCode=0
Jul 18 00:42:02.735813 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.735752    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890"}
Jul 18 00:42:02.738036 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.737998    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mtpf" event={"ID":"6e282b8c-73e9-491f-bfa5-46be537dd4d4","Type":"ContainerStarted","Data":"90f6076ec3f1850fc38a03f6573485231f340a64686b010f120943b5188d78a6"}
Jul 18 00:42:02.738137 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.738048    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mtpf" event={"ID":"6e282b8c-73e9-491f-bfa5-46be537dd4d4","Type":"ContainerStarted","Data":"21e8d1a99744b5a4fa0066e5319f6e4a47734587e8fbb982eeec7fc63e201164"}
Jul 18 00:42:02.740411 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.740392    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k" event={"ID":"8c77e01c-9265-452d-b94d-30c8eb6a6b56","Type":"ContainerStarted","Data":"1efa695ebd5bb1e3c80a41088d9a0e159f1b263cadf412e5cc264d23b1857dee"}
Jul 18 00:42:02.740503 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.740430    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k" event={"ID":"8c77e01c-9265-452d-b94d-30c8eb6a6b56","Type":"ContainerStarted","Data":"052d68f2a58577c8e53946e115d224c9db3657bc2f2487c8e379c1b2a952c177"}
Jul 18 00:42:02.740503 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.740447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k" event={"ID":"8c77e01c-9265-452d-b94d-30c8eb6a6b56","Type":"ContainerStarted","Data":"a4e6b482e8181bfa6b8f0dc7e5c075f7c0ea9fb60adaebdce4493fdd406a399d"}
Jul 18 00:42:02.753449 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.753402    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-tllg7" podStartSLOduration=3.462777527 podStartE2EDuration="5.753386834s" totalImagesPullingTime="2.290609307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:02.751873647 +0000 UTC m=+78.877808625" watchObservedRunningTime="2026-07-18 00:42:02.753386834 +0000 UTC m=+78.879321814"
Jul 18 00:42:02.777445 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.777398    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-x744k" podStartSLOduration=3.1323898039999998 podStartE2EDuration="5.777383955s" totalImagesPullingTime="2.644994151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:02.776011269 +0000 UTC m=+78.901946249" watchObservedRunningTime="2026-07-18 00:42:02.777383955 +0000 UTC m=+78.903319040"
Jul 18 00:42:02.822355 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.822324    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:02.829677 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.829611    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8mtpf" podStartSLOduration=4.85848926 podStartE2EDuration="5.829597226s" totalImagesPullingTime="906.97985ms" totalInitContainerRuntime="64.128116ms" isStatefulPod=true podCreationTimestamp="2026-07-18 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:02.828417326 +0000 UTC m=+78.954352298" watchObservedRunningTime="2026-07-18 00:42:02.829597226 +0000 UTC m=+78.955532205"
Jul 18 00:42:02.970133 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:02.970099    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"]
Jul 18 00:42:02.973431 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:42:02.973400    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e8eece_45ff_4481_b41a_bad92ad7fdad.slice/crio-bce346c276cc42a315010528404148809ac9285e548fc6f264b47d07ed0aa397 WatchSource:0}: Error finding container bce346c276cc42a315010528404148809ac9285e548fc6f264b47d07ed0aa397: Status 404 returned error can't find the container with id bce346c276cc42a315010528404148809ac9285e548fc6f264b47d07ed0aa397
Jul 18 00:42:03.299973 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.299810    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:42:03.309281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.309242    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.312051 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.312019    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 18 00:42:03.312291 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.312274    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 18 00:42:03.312479 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.312462    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 18 00:42:03.312536 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.312488    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 18 00:42:03.312666 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.312650    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 18 00:42:03.313178 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.312849    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 18 00:42:03.313178 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.313056    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 18 00:42:03.313676 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.313468    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 18 00:42:03.313676 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.313528    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 18 00:42:03.313923 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.313843    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 18 00:42:03.318151 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.315786    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6fwzg\""
Jul 18 00:42:03.318151 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.316008    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-64cvpcic80kml\""
Jul 18 00:42:03.318151 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.317903    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 18 00:42:03.318720 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.318665    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 18 00:42:03.319507 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.319037    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 18 00:42:03.324497 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.324467    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 18 00:42:03.326516 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.326492    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:42:03.447356 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447324    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cjp\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-kube-api-access-r6cjp\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447551 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447366    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447551 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447398    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447551 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447464    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447551 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447529    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447556    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447653    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447680    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447725    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.447773 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447769    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447793    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447814    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447918    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.447953    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.448002    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448295 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.448026    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448295 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.448086    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.448295 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.448128    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.549904 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.549863    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550117 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.549920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550117 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.549956    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550117 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550003    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550117 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550033    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cjp\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-kube-api-access-r6cjp\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550117 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550059    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550117 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550091    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550132    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550161    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550186    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550254    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550311    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550338    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550395    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550845 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550418    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550845 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.550845 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.550474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.552580 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.552412    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.552580 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.552456    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.552580 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.552520    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.553725 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.552959    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.556130 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.556101    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.556630 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.556586    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.564708 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.564459    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.565854 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.564987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.565854 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.565293    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.565854 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.565541    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.567639 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.566425    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.567639 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.566968    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.569090 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.568455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cjp\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-kube-api-access-r6cjp\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.569090 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.568991    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.570484 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.570417    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.571263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.570740    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.571263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.570768    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55547d65ff-hc2st"]
Jul 18 00:42:03.571263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.571106    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.571263 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.571214    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.571867 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.571789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.573139 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.573072    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.573511 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.573490    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.629515 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.629413    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:03.748194 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:03.748153    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j" event={"ID":"f7e8eece-45ff-4481-b41a-bad92ad7fdad","Type":"ContainerStarted","Data":"bce346c276cc42a315010528404148809ac9285e548fc6f264b47d07ed0aa397"}
Jul 18 00:42:04.094193 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.094152    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:42:04.100256 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:42:04.100224    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab45a4e_eaac_475a_95a0_72cc098b3d6d.slice/crio-951cb18005062385581255ec62de9ae569677ffa83f8eccd5f3340e661e5de65 WatchSource:0}: Error finding container 951cb18005062385581255ec62de9ae569677ffa83f8eccd5f3340e661e5de65: Status 404 returned error can't find the container with id 951cb18005062385581255ec62de9ae569677ffa83f8eccd5f3340e661e5de65
Jul 18 00:42:04.757357 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.757116    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" event={"ID":"17809a4e-df1b-40b8-9b2f-10f489d32b97","Type":"ContainerStarted","Data":"175801630bd52b756b1de91725c4c62fc53ce47f8e10ea6df466924c4f8b7434"}
Jul 18 00:42:04.757357 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.757197    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" event={"ID":"17809a4e-df1b-40b8-9b2f-10f489d32b97","Type":"ContainerStarted","Data":"9167c92aaf940425563421605279680e103ca29b5212649c2ee7c3c6ae764946"}
Jul 18 00:42:04.757357 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.757240    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" event={"ID":"17809a4e-df1b-40b8-9b2f-10f489d32b97","Type":"ContainerStarted","Data":"0d4fb122ef4c50c3d74632df40d1721e7995a14086273fa4647531dd7da2d624"}
Jul 18 00:42:04.757923 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.757409    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:42:04.758966 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.758938    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6" exitCode=0
Jul 18 00:42:04.759094 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.759035    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"}
Jul 18 00:42:04.759163 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.759109    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerStarted","Data":"951cb18005062385581255ec62de9ae569677ffa83f8eccd5f3340e661e5de65"}
Jul 18 00:42:04.761664 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.761595    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk" event={"ID":"a2040fa1-7916-4610-bfcb-445d2faa646e","Type":"ContainerStarted","Data":"91afd70d2db4513c5eb2e8bd8b43f4cad5631d5202e1af58035909b8deaf926c"}
Jul 18 00:42:04.780370 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.780310    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7" podStartSLOduration=1.694384254 podStartE2EDuration="5.780290125s" totalImagesPullingTime="4.085905871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:04.778922611 +0000 UTC m=+80.904857593" watchObservedRunningTime="2026-07-18 00:42:04.780290125 +0000 UTC m=+80.906225106"
Jul 18 00:42:04.797896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:04.797845    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk" podStartSLOduration=2.075377161 podStartE2EDuration="3.797829193s" totalImagesPullingTime="1.722452032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:04.796412121 +0000 UTC m=+80.922347108" watchObservedRunningTime="2026-07-18 00:42:04.797829193 +0000 UTC m=+80.923764172"
Jul 18 00:42:05.647430 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.647354    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:42:05.768331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.768293    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j" event={"ID":"f7e8eece-45ff-4481-b41a-bad92ad7fdad","Type":"ContainerStarted","Data":"f267cf3869a8e4c01b0a38196027f4511c567f3e86cc05cd762af509c5852705"}
Jul 18 00:42:05.768769 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.768359    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:05.771885 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.771856    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerStarted","Data":"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"}
Jul 18 00:42:05.772031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.771934    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerStarted","Data":"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"}
Jul 18 00:42:05.772031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.771978    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerStarted","Data":"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"}
Jul 18 00:42:05.772031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.772017    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerStarted","Data":"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"}
Jul 18 00:42:05.772188 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.772056    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerStarted","Data":"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"}
Jul 18 00:42:05.772188 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.772095    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerStarted","Data":"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"}
Jul 18 00:42:05.773839 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.773818    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j"
Jul 18 00:42:05.788964 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.788917    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-8wj2j" podStartSLOduration=2.673981238 podStartE2EDuration="4.788902321s" totalImagesPullingTime="2.114921083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:05.787138206 +0000 UTC m=+81.913073186" watchObservedRunningTime="2026-07-18 00:42:05.788902321 +0000 UTC m=+81.914837301"
Jul 18 00:42:05.818898 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:05.818848    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.038753561 podStartE2EDuration="7.818832536s" totalImagesPullingTime="4.745400296s" totalInitContainerRuntime="34.678679ms" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:05.816928305 +0000 UTC m=+81.942863315" watchObservedRunningTime="2026-07-18 00:42:05.818832536 +0000 UTC m=+81.944767514"
Jul 18 00:42:08.792942 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:08.792909    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerStarted","Data":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"}
Jul 18 00:42:08.793345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:08.792974    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerStarted","Data":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"}
Jul 18 00:42:08.793345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:08.793001    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerStarted","Data":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"}
Jul 18 00:42:08.793345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:08.793034    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerStarted","Data":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"}
Jul 18 00:42:08.793345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:08.793058    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerStarted","Data":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"}
Jul 18 00:42:08.793345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:08.793080    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerStarted","Data":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"}
Jul 18 00:42:10.778685 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:10.778657    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b987bb95b-mctr7"
Jul 18 00:42:10.802632 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:10.802574    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.562747634 podStartE2EDuration="7.802556226s" totalImagesPullingTime="3.210235774s" totalInitContainerRuntime="29.572818ms" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:08.825113067 +0000 UTC m=+84.951048046" watchObservedRunningTime="2026-07-18 00:42:10.802556226 +0000 UTC m=+86.928491269"
Jul 18 00:42:13.630362 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:13.630322    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:42:14.881197 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:14.881159    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4fdd577b-f4x9g"]
Jul 18 00:42:21.801646 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:21.801598    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:21.802124 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:21.801670    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:28.600478 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:28.600417    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55547d65ff-hc2st" podUID="d338c2f8-0ced-4a1a-9caf-caf99d335d0b" containerName="console" containerID="cri-o://c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8" gracePeriod=15
Jul 18 00:42:29.372964 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.372933    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:42:29.497101 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497069    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-serving-cert\") pod \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") "
Jul 18 00:42:29.497101 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497112    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-config\") pod \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") "
Jul 18 00:42:29.497356 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497175    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-oauth-serving-cert\") pod \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") "
Jul 18 00:42:29.497356 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497202    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-trusted-ca-bundle\") pod \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") "
Jul 18 00:42:29.497356 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497338    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-oauth-config\") pod \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") "
Jul 18 00:42:29.497512 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497392    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-service-ca\") pod \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") "
Jul 18 00:42:29.497512 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497448    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5ksf\" (UniqueName: \"kubernetes.io/projected/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-kube-api-access-q5ksf\") pod \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\" (UID: \"d338c2f8-0ced-4a1a-9caf-caf99d335d0b\") "
Jul 18 00:42:29.497767 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497730    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-config" (OuterVolumeSpecName: "console-config") pod "d338c2f8-0ced-4a1a-9caf-caf99d335d0b" (UID: "d338c2f8-0ced-4a1a-9caf-caf99d335d0b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:29.497885 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497776    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d338c2f8-0ced-4a1a-9caf-caf99d335d0b" (UID: "d338c2f8-0ced-4a1a-9caf-caf99d335d0b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:29.497885 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497791    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d338c2f8-0ced-4a1a-9caf-caf99d335d0b" (UID: "d338c2f8-0ced-4a1a-9caf-caf99d335d0b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:29.497972 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.497906    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-service-ca" (OuterVolumeSpecName: "service-ca") pod "d338c2f8-0ced-4a1a-9caf-caf99d335d0b" (UID: "d338c2f8-0ced-4a1a-9caf-caf99d335d0b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:29.499537 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.499513    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d338c2f8-0ced-4a1a-9caf-caf99d335d0b" (UID: "d338c2f8-0ced-4a1a-9caf-caf99d335d0b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:42:29.499671 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.499580    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-kube-api-access-q5ksf" (OuterVolumeSpecName: "kube-api-access-q5ksf") pod "d338c2f8-0ced-4a1a-9caf-caf99d335d0b" (UID: "d338c2f8-0ced-4a1a-9caf-caf99d335d0b"). InnerVolumeSpecName "kube-api-access-q5ksf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:42:29.499671 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.499601    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d338c2f8-0ced-4a1a-9caf-caf99d335d0b" (UID: "d338c2f8-0ced-4a1a-9caf-caf99d335d0b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:42:29.599166 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.599080    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-oauth-serving-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:29.599166 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.599114    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-trusted-ca-bundle\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:29.599166 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.599126    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-oauth-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:29.599166 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.599138    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:29.599166 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.599149    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ksf\" (UniqueName: \"kubernetes.io/projected/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-kube-api-access-q5ksf\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:29.599166 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.599162    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-serving-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:29.599526 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.599175    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d338c2f8-0ced-4a1a-9caf-caf99d335d0b-console-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:29.901271 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.901190    2554 generic.go:356] "Generic (PLEG): container finished" podID="d338c2f8-0ced-4a1a-9caf-caf99d335d0b" containerID="c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8" exitCode=0
Jul 18 00:42:29.901271 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.901249    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55547d65ff-hc2st"
Jul 18 00:42:29.901716 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.901272    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55547d65ff-hc2st" event={"ID":"d338c2f8-0ced-4a1a-9caf-caf99d335d0b","Type":"ContainerDied","Data":"c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8"}
Jul 18 00:42:29.901716 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.901342    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55547d65ff-hc2st" event={"ID":"d338c2f8-0ced-4a1a-9caf-caf99d335d0b","Type":"ContainerDied","Data":"bc82be9b2a1651275ba0a180812d7dcd34263deda2378b7f094c99c012c0eb60"}
Jul 18 00:42:29.901716 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.901363    2554 scope.go:117] "RemoveContainer" containerID="c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8"
Jul 18 00:42:29.912201 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.912186    2554 scope.go:117] "RemoveContainer" containerID="c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8"
Jul 18 00:42:29.912454 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:42:29.912435    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8\": container with ID starting with c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8 not found: ID does not exist" containerID="c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8"
Jul 18 00:42:29.912509 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.912463    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8"} err="failed to get container status \"c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8\": rpc error: code = NotFound desc = could not find container \"c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8\": container with ID starting with c4b4d378408b1debd0429ce25e75a8499c11e5657573371dd3303d68a14ba6d8 not found: ID does not exist"
Jul 18 00:42:29.922312 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.922293    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55547d65ff-hc2st"]
Jul 18 00:42:29.926069 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:29.926052    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55547d65ff-hc2st"]
Jul 18 00:42:30.335531 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:30.335487    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d338c2f8-0ced-4a1a-9caf-caf99d335d0b" path="/var/lib/kubelet/pods/d338c2f8-0ced-4a1a-9caf-caf99d335d0b/volumes"
Jul 18 00:42:30.719075 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:30.718998    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qqqj9"
Jul 18 00:42:39.902009 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:39.901964    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c4fdd577b-f4x9g" podUID="bc6be778-f6b7-485e-8e76-021a4dc37e95" containerName="console" containerID="cri-o://e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055" gracePeriod=15
Jul 18 00:42:40.145785 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.145755    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:42:40.295005 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.294973    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvk68\" (UniqueName: \"kubernetes.io/projected/bc6be778-f6b7-485e-8e76-021a4dc37e95-kube-api-access-jvk68\") pod \"bc6be778-f6b7-485e-8e76-021a4dc37e95\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") "
Jul 18 00:42:40.295187 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295024    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-service-ca\") pod \"bc6be778-f6b7-485e-8e76-021a4dc37e95\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") "
Jul 18 00:42:40.295187 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295046    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-serving-cert\") pod \"bc6be778-f6b7-485e-8e76-021a4dc37e95\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") "
Jul 18 00:42:40.295300 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295179    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-oauth-serving-cert\") pod \"bc6be778-f6b7-485e-8e76-021a4dc37e95\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") "
Jul 18 00:42:40.295300 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295291    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-oauth-config\") pod \"bc6be778-f6b7-485e-8e76-021a4dc37e95\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") "
Jul 18 00:42:40.295396 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295321    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-config\") pod \"bc6be778-f6b7-485e-8e76-021a4dc37e95\" (UID: \"bc6be778-f6b7-485e-8e76-021a4dc37e95\") "
Jul 18 00:42:40.295484 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295370    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc6be778-f6b7-485e-8e76-021a4dc37e95" (UID: "bc6be778-f6b7-485e-8e76-021a4dc37e95"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:40.295561 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295527    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc6be778-f6b7-485e-8e76-021a4dc37e95" (UID: "bc6be778-f6b7-485e-8e76-021a4dc37e95"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:40.295745 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295724    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:40.295812 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295749    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-oauth-serving-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:40.295812 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.295755    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-config" (OuterVolumeSpecName: "console-config") pod "bc6be778-f6b7-485e-8e76-021a4dc37e95" (UID: "bc6be778-f6b7-485e-8e76-021a4dc37e95"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:40.297320 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.297295    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc6be778-f6b7-485e-8e76-021a4dc37e95" (UID: "bc6be778-f6b7-485e-8e76-021a4dc37e95"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:42:40.297421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.297403    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6be778-f6b7-485e-8e76-021a4dc37e95-kube-api-access-jvk68" (OuterVolumeSpecName: "kube-api-access-jvk68") pod "bc6be778-f6b7-485e-8e76-021a4dc37e95" (UID: "bc6be778-f6b7-485e-8e76-021a4dc37e95"). InnerVolumeSpecName "kube-api-access-jvk68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:42:40.297421 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.297408    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc6be778-f6b7-485e-8e76-021a4dc37e95" (UID: "bc6be778-f6b7-485e-8e76-021a4dc37e95"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:42:40.396204 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.396170    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvk68\" (UniqueName: \"kubernetes.io/projected/bc6be778-f6b7-485e-8e76-021a4dc37e95-kube-api-access-jvk68\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:40.396204 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.396199    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-serving-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:40.396204 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.396209    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-oauth-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:40.396428 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.396220    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc6be778-f6b7-485e-8e76-021a4dc37e95-console-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:42:40.947423 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.947388    2554 generic.go:356] "Generic (PLEG): container finished" podID="bc6be778-f6b7-485e-8e76-021a4dc37e95" containerID="e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055" exitCode=0
Jul 18 00:42:40.947934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.947440    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fdd577b-f4x9g"
Jul 18 00:42:40.947934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.947475    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fdd577b-f4x9g" event={"ID":"bc6be778-f6b7-485e-8e76-021a4dc37e95","Type":"ContainerDied","Data":"e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055"}
Jul 18 00:42:40.947934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.947539    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fdd577b-f4x9g" event={"ID":"bc6be778-f6b7-485e-8e76-021a4dc37e95","Type":"ContainerDied","Data":"7d5c9f7329e61a17bf26f3a7f108333a6cc865535ec89958f704c015939bb43e"}
Jul 18 00:42:40.947934 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.947560    2554 scope.go:117] "RemoveContainer" containerID="e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055"
Jul 18 00:42:40.955892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.955877    2554 scope.go:117] "RemoveContainer" containerID="e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055"
Jul 18 00:42:40.956143 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:42:40.956122    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055\": container with ID starting with e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055 not found: ID does not exist" containerID="e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055"
Jul 18 00:42:40.956206 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.956150    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055"} err="failed to get container status \"e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055\": rpc error: code = NotFound desc = could not find container \"e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055\": container with ID starting with e376207aceabd2a6ed79b974a70b42d4373929d8627a095238ba90cff1758055 not found: ID does not exist"
Jul 18 00:42:40.964202 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.964151    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4fdd577b-f4x9g"]
Jul 18 00:42:40.965626 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:40.965595    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c4fdd577b-f4x9g"]
Jul 18 00:42:41.807468 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:41.807439    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:41.811336 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:41.811315    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c5ccd67dc-wk5wk"
Jul 18 00:42:42.336112 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:42:42.336077    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6be778-f6b7-485e-8e76-021a4dc37e95" path="/var/lib/kubelet/pods/bc6be778-f6b7-485e-8e76-021a4dc37e95/volumes"
Jul 18 00:43:03.629790 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:03.629753    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:03.648187 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:03.648162    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:04.068289 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:04.068262    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:17.331086 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:17.331049    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:43:17.331630 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:17.331307    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="alertmanager" containerID="cri-o://d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5" gracePeriod=120
Jul 18 00:43:17.331630 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:17.331362    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-web" containerID="cri-o://4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939" gracePeriod=120
Jul 18 00:43:17.331630 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:17.331362    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-metric" containerID="cri-o://6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92" gracePeriod=120
Jul 18 00:43:17.331630 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:17.331410    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy" containerID="cri-o://a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8" gracePeriod=120
Jul 18 00:43:17.331630 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:17.331459    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="prom-label-proxy" containerID="cri-o://39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51" gracePeriod=120
Jul 18 00:43:17.331630 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:17.331477    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="config-reloader" containerID="cri-o://2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3" gracePeriod=120
Jul 18 00:43:18.108743 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108708    2554 generic.go:356] "Generic (PLEG): container finished" podID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerID="39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51" exitCode=0
Jul 18 00:43:18.108743 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108744    2554 generic.go:356] "Generic (PLEG): container finished" podID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerID="a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8" exitCode=0
Jul 18 00:43:18.108743 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108751    2554 generic.go:356] "Generic (PLEG): container finished" podID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerID="2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3" exitCode=0
Jul 18 00:43:18.108976 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108760    2554 generic.go:356] "Generic (PLEG): container finished" podID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerID="d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5" exitCode=0
Jul 18 00:43:18.108976 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108781    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"}
Jul 18 00:43:18.108976 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108873    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"}
Jul 18 00:43:18.108976 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"}
Jul 18 00:43:18.108976 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.108924    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"}
Jul 18 00:43:18.575776 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.575752    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:18.639727 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.639689    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-trusted-ca-bundle\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.639879 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.639813    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-cluster-tls-config\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.639879 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.639848    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-main-tls\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.639992 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.639900    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.639992 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.639930    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-web-config\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.639992 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.639961    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-tls-assets\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640141 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.639994    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-config-out\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640195 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640137    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:18.640403 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640380    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640481 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640431    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2chqg\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-kube-api-access-2chqg\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640533 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640477    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640533 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640513    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-config-volume\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640665 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640540    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-metrics-client-ca\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640665 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640575    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-main-db\") pod \"922e1204-62c1-4e57-9a71-7b9f259139a9\" (UID: \"922e1204-62c1-4e57-9a71-7b9f259139a9\") "
Jul 18 00:43:18.640900 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.640878    2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.641671 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.641641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:43:18.642921 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.642541    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:18.644100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.644070    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:18.644555 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.644444    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-config-out" (OuterVolumeSpecName: "config-out") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:43:18.644555 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.644528    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:43:18.644888 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.644861    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:18.645054 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.645034    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:18.645776 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.645743    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-config-volume" (OuterVolumeSpecName: "config-volume") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:18.645902 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.645868    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-kube-api-access-2chqg" (OuterVolumeSpecName: "kube-api-access-2chqg") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "kube-api-access-2chqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:43:18.646601 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.646577    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:18.649674 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.649652    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:18.655517 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.655496    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-web-config" (OuterVolumeSpecName: "web-config") pod "922e1204-62c1-4e57-9a71-7b9f259139a9" (UID: "922e1204-62c1-4e57-9a71-7b9f259139a9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:18.741534 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741498    2554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-cluster-tls-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741534 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741525    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-main-tls\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741534 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741537    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741547    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-web-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741555    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-tls-assets\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741563    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-config-out\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741571    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741579    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chqg\" (UniqueName: \"kubernetes.io/projected/922e1204-62c1-4e57-9a71-7b9f259139a9-kube-api-access-2chqg\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741588    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741596    2554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/922e1204-62c1-4e57-9a71-7b9f259139a9-config-volume\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741604    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922e1204-62c1-4e57-9a71-7b9f259139a9-metrics-client-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:18.741848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:18.741631    2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/922e1204-62c1-4e57-9a71-7b9f259139a9-alertmanager-main-db\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:19.117501 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.117468    2554 generic.go:356] "Generic (PLEG): container finished" podID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerID="6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92" exitCode=0
Jul 18 00:43:19.117501 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.117492    2554 generic.go:356] "Generic (PLEG): container finished" podID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerID="4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939" exitCode=0
Jul 18 00:43:19.117712 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.117536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"}
Jul 18 00:43:19.117712 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.117545    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.117712 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.117602    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"}
Jul 18 00:43:19.117712 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.117648    2554 scope.go:117] "RemoveContainer" containerID="39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"
Jul 18 00:43:19.117712 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.117667    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"922e1204-62c1-4e57-9a71-7b9f259139a9","Type":"ContainerDied","Data":"139742fff601f09e72f00db61721eacb966e073fda21e2aeacedf2a9b21a489c"}
Jul 18 00:43:19.125410 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.125392    2554 scope.go:117] "RemoveContainer" containerID="6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"
Jul 18 00:43:19.132271 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.132254    2554 scope.go:117] "RemoveContainer" containerID="a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"
Jul 18 00:43:19.138240 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.138216    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:43:19.140318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.140255    2554 scope.go:117] "RemoveContainer" containerID="4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"
Jul 18 00:43:19.141966 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.141947    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:43:19.147277 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.147261    2554 scope.go:117] "RemoveContainer" containerID="2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"
Jul 18 00:43:19.153982 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.153967    2554 scope.go:117] "RemoveContainer" containerID="d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"
Jul 18 00:43:19.161217 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.161200    2554 scope.go:117] "RemoveContainer" containerID="c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890"
Jul 18 00:43:19.166318 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166298    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:43:19.166732 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166714    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-web"
Jul 18 00:43:19.166732 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166732    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-web"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166743    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="alertmanager"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166749    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="alertmanager"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166761    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="config-reloader"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166768    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="config-reloader"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166775    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6be778-f6b7-485e-8e76-021a4dc37e95" containerName="console"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166783    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6be778-f6b7-485e-8e76-021a4dc37e95" containerName="console"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166793    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="init-config-reloader"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166798    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="init-config-reloader"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166811    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d338c2f8-0ced-4a1a-9caf-caf99d335d0b" containerName="console"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166818    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d338c2f8-0ced-4a1a-9caf-caf99d335d0b" containerName="console"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166830    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="prom-label-proxy"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166836    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="prom-label-proxy"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166848    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy"
Jul 18 00:43:19.166849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166853    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166869    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-metric"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166876    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-metric"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166925    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="config-reloader"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166933    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="alertmanager"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166940    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-web"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166949    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d338c2f8-0ced-4a1a-9caf-caf99d335d0b" containerName="console"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166955    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy-metric"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166961    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="prom-label-proxy"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166966    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" containerName="kube-rbac-proxy"
Jul 18 00:43:19.167261 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.166972    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6be778-f6b7-485e-8e76-021a4dc37e95" containerName="console"
Jul 18 00:43:19.169288 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.169272    2554 scope.go:117] "RemoveContainer" containerID="39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"
Jul 18 00:43:19.169548 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:19.169530    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51\": container with ID starting with 39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51 not found: ID does not exist" containerID="39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"
Jul 18 00:43:19.169632 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.169559    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"} err="failed to get container status \"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51\": rpc error: code = NotFound desc = could not find container \"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51\": container with ID starting with 39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51 not found: ID does not exist"
Jul 18 00:43:19.169632 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.169579    2554 scope.go:117] "RemoveContainer" containerID="6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"
Jul 18 00:43:19.169887 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:19.169872    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92\": container with ID starting with 6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92 not found: ID does not exist" containerID="6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"
Jul 18 00:43:19.169926 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.169892    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"} err="failed to get container status \"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92\": rpc error: code = NotFound desc = could not find container \"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92\": container with ID starting with 6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92 not found: ID does not exist"
Jul 18 00:43:19.169926 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.169908    2554 scope.go:117] "RemoveContainer" containerID="a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"
Jul 18 00:43:19.170162 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:19.170143    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8\": container with ID starting with a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8 not found: ID does not exist" containerID="a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"
Jul 18 00:43:19.170205 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170169    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"} err="failed to get container status \"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8\": rpc error: code = NotFound desc = could not find container \"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8\": container with ID starting with a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8 not found: ID does not exist"
Jul 18 00:43:19.170205 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170185    2554 scope.go:117] "RemoveContainer" containerID="4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"
Jul 18 00:43:19.170398 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:19.170381    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939\": container with ID starting with 4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939 not found: ID does not exist" containerID="4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"
Jul 18 00:43:19.170463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170404    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"} err="failed to get container status \"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939\": rpc error: code = NotFound desc = could not find container \"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939\": container with ID starting with 4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939 not found: ID does not exist"
Jul 18 00:43:19.170463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170425    2554 scope.go:117] "RemoveContainer" containerID="2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"
Jul 18 00:43:19.170667 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:19.170652    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3\": container with ID starting with 2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3 not found: ID does not exist" containerID="2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"
Jul 18 00:43:19.170720 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170671    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"} err="failed to get container status \"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3\": rpc error: code = NotFound desc = could not find container \"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3\": container with ID starting with 2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3 not found: ID does not exist"
Jul 18 00:43:19.170720 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170685    2554 scope.go:117] "RemoveContainer" containerID="d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"
Jul 18 00:43:19.170915 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:19.170893    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5\": container with ID starting with d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5 not found: ID does not exist" containerID="d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"
Jul 18 00:43:19.171015 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170915    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"} err="failed to get container status \"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5\": rpc error: code = NotFound desc = could not find container \"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5\": container with ID starting with d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5 not found: ID does not exist"
Jul 18 00:43:19.171015 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.170929    2554 scope.go:117] "RemoveContainer" containerID="c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890"
Jul 18 00:43:19.171312 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:19.171287    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890\": container with ID starting with c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890 not found: ID does not exist" containerID="c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890"
Jul 18 00:43:19.171372 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.171333    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890"} err="failed to get container status \"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890\": rpc error: code = NotFound desc = could not find container \"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890\": container with ID starting with c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890 not found: ID does not exist"
Jul 18 00:43:19.171372 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.171351    2554 scope.go:117] "RemoveContainer" containerID="39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"
Jul 18 00:43:19.171722 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.171610    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51"} err="failed to get container status \"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51\": rpc error: code = NotFound desc = could not find container \"39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51\": container with ID starting with 39844a701a0007ac930199ef71497309600ae9735241669ed9b6ae175f452e51 not found: ID does not exist"
Jul 18 00:43:19.171722 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.171652    2554 scope.go:117] "RemoveContainer" containerID="6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"
Jul 18 00:43:19.172326 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.172306    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.173912 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.173889    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92"} err="failed to get container status \"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92\": rpc error: code = NotFound desc = could not find container \"6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92\": container with ID starting with 6a4f3e0e4f8b07debd92985a70f68fb0905ed8ca9330d3a1e6168619e09bfb92 not found: ID does not exist"
Jul 18 00:43:19.173912 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.173911    2554 scope.go:117] "RemoveContainer" containerID="a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"
Jul 18 00:43:19.174155 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174138    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8"} err="failed to get container status \"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8\": rpc error: code = NotFound desc = could not find container \"a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8\": container with ID starting with a890dd88a49b0cffacf81929e391b79fa64db19a1dc15b36607ced8c8709cff8 not found: ID does not exist"
Jul 18 00:43:19.174155 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174154    2554 scope.go:117] "RemoveContainer" containerID="4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"
Jul 18 00:43:19.174401 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174382    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939"} err="failed to get container status \"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939\": rpc error: code = NotFound desc = could not find container \"4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939\": container with ID starting with 4694858877fa0be9ed55b6ad267e7f5562967480f64a86c765eab1776dd31939 not found: ID does not exist"
Jul 18 00:43:19.174401 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174400    2554 scope.go:117] "RemoveContainer" containerID="2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"
Jul 18 00:43:19.174669 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174649    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3"} err="failed to get container status \"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3\": rpc error: code = NotFound desc = could not find container \"2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3\": container with ID starting with 2e0f4fa14bf85ac3e53de40a4401ad9d66a241dd61babb0e42b319ede7b83ff3 not found: ID does not exist"
Jul 18 00:43:19.174669 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174669    2554 scope.go:117] "RemoveContainer" containerID="d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"
Jul 18 00:43:19.174887 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174867    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 18 00:43:19.174986 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174959    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5"} err="failed to get container status \"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5\": rpc error: code = NotFound desc = could not find container \"d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5\": container with ID starting with d6dc5a79c31659f99511a899d04c8cc056d2bfac04c75217c134037812d741c5 not found: ID does not exist"
Jul 18 00:43:19.175033 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.174990    2554 scope.go:117] "RemoveContainer" containerID="c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890"
Jul 18 00:43:19.175120 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175103    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jts4m\""
Jul 18 00:43:19.175243 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175224    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890"} err="failed to get container status \"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890\": rpc error: code = NotFound desc = could not find container \"c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890\": container with ID starting with c97ac2de45f8f732d5981fef061804f3bfcd3f80674588f0e4713840e91e0890 not found: ID does not exist"
Jul 18 00:43:19.175397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175383    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 18 00:43:19.175458 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175432    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 18 00:43:19.175781 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175766    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 18 00:43:19.175877 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175766    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 18 00:43:19.175877 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175822    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 18 00:43:19.175877 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175819    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 18 00:43:19.176022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.175857    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 18 00:43:19.182211 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.182182    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 18 00:43:19.183361 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.183340    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:43:19.246861 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.246826    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.246861 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.246874    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvvm\" (UniqueName: \"kubernetes.io/projected/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-kube-api-access-xvvvm\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247081 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.246948    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247081 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-web-config\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247081 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247024    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247081 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247045    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247229 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247229 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247164    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-config-out\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247229 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247181    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247229 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247353 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247241    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247353 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247258    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.247353 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.247282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348129 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348050    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-config-out\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348129 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348083    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348129 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348105    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348129 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348129    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348145    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348207    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348249    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348269    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvvm\" (UniqueName: \"kubernetes.io/projected/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-kube-api-access-xvvvm\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348309    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-web-config\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348375    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348427 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348846 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.348846 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.348484    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.349213 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.349190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351435 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351213    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-config-out\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351435 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351245    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351435 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351371    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351435 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351380    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-web-config\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351756 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351525    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-config-volume\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351756 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351538    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351756 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.351869 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.351759    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.352072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.352054    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.353268 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.353247    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.359148 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.359126    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvvm\" (UniqueName: \"kubernetes.io/projected/a4a8b40e-7a11-482f-b9fb-27cab2d81aa8-kube-api-access-xvvvm\") pod \"alertmanager-main-0\" (UID: \"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.483392 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.483358    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 18 00:43:19.618598 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:19.618572    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 18 00:43:19.620367 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:43:19.620335    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a8b40e_7a11_482f_b9fb_27cab2d81aa8.slice/crio-c74260a20de550eb6e2fdad98b572b114538717e0afbba3e6bb3f110346e5b5c WatchSource:0}: Error finding container c74260a20de550eb6e2fdad98b572b114538717e0afbba3e6bb3f110346e5b5c: Status 404 returned error can't find the container with id c74260a20de550eb6e2fdad98b572b114538717e0afbba3e6bb3f110346e5b5c
Jul 18 00:43:20.123871 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:20.123830    2554 generic.go:356] "Generic (PLEG): container finished" podID="a4a8b40e-7a11-482f-b9fb-27cab2d81aa8" containerID="8b8d6cc87afdaa791be64a7a63dc5ef6e313c11f209a2952db7b8a09fb7757d3" exitCode=0
Jul 18 00:43:20.124031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:20.123904    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerDied","Data":"8b8d6cc87afdaa791be64a7a63dc5ef6e313c11f209a2952db7b8a09fb7757d3"}
Jul 18 00:43:20.124031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:20.124006    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerStarted","Data":"c74260a20de550eb6e2fdad98b572b114538717e0afbba3e6bb3f110346e5b5c"}
Jul 18 00:43:20.336710 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:20.336684    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922e1204-62c1-4e57-9a71-7b9f259139a9" path="/var/lib/kubelet/pods/922e1204-62c1-4e57-9a71-7b9f259139a9/volumes"
Jul 18 00:43:21.131374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.131340    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerStarted","Data":"a16011856f7869a626e88af509e2b3484fd2dd4d32b04423333577f7f673fe3f"}
Jul 18 00:43:21.131892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.131415    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerStarted","Data":"3b2d1a9238a6118df74a2276208c8dbaeb04dc21481ce6c98e1935436efd51e9"}
Jul 18 00:43:21.131892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.131462    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerStarted","Data":"91c0ba6376953da7a31faa125b31aaf2e6e2bc744159c0ee141b3d1428c89a19"}
Jul 18 00:43:21.131892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.131512    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerStarted","Data":"f98c1f8653213c7da3df84e3ef7badc9a4c0344a414db88a4694c42308ed10a6"}
Jul 18 00:43:21.131892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.131556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerStarted","Data":"7f655f4d3601815c15c1721eb5e63c184ca87381116062886a50982a6e5f66c4"}
Jul 18 00:43:21.131892 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.131590    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a4a8b40e-7a11-482f-b9fb-27cab2d81aa8","Type":"ContainerStarted","Data":"8142e4ed050000f479db7e1251e5ee69a95769d5fcdcdb99e2c852d299bc80c4"}
Jul 18 00:43:21.156248 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.156159    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.139041579 podStartE2EDuration="2.156142159s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.10058ms" isStatefulPod=false podCreationTimestamp="2026-07-18 00:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:43:21.154036166 +0000 UTC m=+157.279971148" watchObservedRunningTime="2026-07-18 00:43:21.156142159 +0000 UTC m=+157.282077139"
Jul 18 00:43:21.672641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.670404    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:43:21.672641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.671020    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="prometheus" containerID="cri-o://4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011" gracePeriod=600
Jul 18 00:43:21.672641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.671287    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy" containerID="cri-o://141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362" gracePeriod=600
Jul 18 00:43:21.672641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.671451    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-web" containerID="cri-o://91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30" gracePeriod=600
Jul 18 00:43:21.672641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.671505    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f" gracePeriod=600
Jul 18 00:43:21.672641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.671569    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="config-reloader" containerID="cri-o://91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e" gracePeriod=600
Jul 18 00:43:21.672641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.671649    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="thanos-sidecar" containerID="cri-o://742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c" gracePeriod=600
Jul 18 00:43:21.933794 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:21.933732    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.072423 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072388    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-kubelet-serving-ca-bundle\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072423 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072425    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-metrics-client-certs\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072678 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072465    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072678 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072492    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-2\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072678 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072517    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-0\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072678 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072551    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config-out\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072678 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072606    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-metrics-client-ca\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072692    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-trusted-ca-bundle\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072719    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-tls-assets\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072747    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-serving-certs-ca-bundle\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072774    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072805    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-grpc-tls\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072830    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-kube-rbac-proxy\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072836    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:22.072917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072852    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-1\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072929    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072949    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072962    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-thanos-prometheus-http-client-file\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.072990    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-tls\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073025    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6cjp\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-kube-api-access-r6cjp\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073058    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-web-config\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073080    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073102    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-db\") pod \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\" (UID: \"2ab45a4e-eaac-475a-95a0-72cc098b3d6d\") "
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073155    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:22.073374 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073232    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:22.073890 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073394    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.073890 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073412    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.073890 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073428    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-metrics-client-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.073890 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073442    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.073890 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073458    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.073890 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073511    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:22.073890 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.073820    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:43:22.075692 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.075643    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.076439 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.076414    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.076439 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.076414    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:43:22.077260 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077126    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.077260 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077218    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-kube-api-access-r6cjp" (OuterVolumeSpecName: "kube-api-access-r6cjp") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "kube-api-access-r6cjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:43:22.077260 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077246    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:43:22.077494 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077266    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.077494 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077282    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.077494 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077321    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.077682 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077564    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config-out" (OuterVolumeSpecName: "config-out") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:43:22.077793 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.077760    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.078097 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.078077    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config" (OuterVolumeSpecName: "config") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.089552 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.089527    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-web-config" (OuterVolumeSpecName: "web-config") pod "2ab45a4e-eaac-475a-95a0-72cc098b3d6d" (UID: "2ab45a4e-eaac-475a-95a0-72cc098b3d6d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:43:22.137638 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137574    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f" exitCode=0
Jul 18 00:43:22.137638 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137603    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362" exitCode=0
Jul 18 00:43:22.137638 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137610    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30" exitCode=0
Jul 18 00:43:22.137638 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137637    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c" exitCode=0
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137658    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e" exitCode=0
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137648    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"}
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137667    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011" exitCode=0
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137671    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137733    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"}
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137748    2554 scope.go:117] "RemoveContainer" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137782    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"}
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137822    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"}
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137864    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"}
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137903    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"}
Jul 18 00:43:22.138331 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.137944    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ab45a4e-eaac-475a-95a0-72cc098b3d6d","Type":"ContainerDied","Data":"951cb18005062385581255ec62de9ae569677ffa83f8eccd5f3340e661e5de65"}
Jul 18 00:43:22.145366 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.145350    2554 scope.go:117] "RemoveContainer" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.152365 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.152346    2554 scope.go:117] "RemoveContainer" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.159471 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.159409    2554 scope.go:117] "RemoveContainer" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.160936 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.160918    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:43:22.165387 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.165362    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:43:22.167363 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.167349    2554 scope.go:117] "RemoveContainer" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.174282 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174265    2554 scope.go:117] "RemoveContainer" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.174657 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174635    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174660    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-tls-assets\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174672    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174680    2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-grpc-tls\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174689    2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-kube-rbac-proxy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174699    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174708    2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174716    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174724    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6cjp\" (UniqueName: \"kubernetes.io/projected/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-kube-api-access-r6cjp\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174732    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-web-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174740    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-db\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174749    2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-secret-metrics-client-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.174754 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174757    2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.175215 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174766    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.175215 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.174777    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab45a4e-eaac-475a-95a0-72cc098b3d6d-config-out\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 00:43:22.181546 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.181531    2554 scope.go:117] "RemoveContainer" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.188211 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.188195    2554 scope.go:117] "RemoveContainer" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.188486 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:22.188464    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": container with ID starting with 460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f not found: ID does not exist" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.188566 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.188501    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"} err="failed to get container status \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": rpc error: code = NotFound desc = could not find container \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": container with ID starting with 460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f not found: ID does not exist"
Jul 18 00:43:22.188566 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.188528    2554 scope.go:117] "RemoveContainer" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.188954 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:22.188926    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": container with ID starting with 141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362 not found: ID does not exist" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.189044 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.188964    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"} err="failed to get container status \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": rpc error: code = NotFound desc = could not find container \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": container with ID starting with 141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362 not found: ID does not exist"
Jul 18 00:43:22.189044 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.188990    2554 scope.go:117] "RemoveContainer" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.189531 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:22.189510    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": container with ID starting with 91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30 not found: ID does not exist" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.189648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.189541    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"} err="failed to get container status \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": rpc error: code = NotFound desc = could not find container \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": container with ID starting with 91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30 not found: ID does not exist"
Jul 18 00:43:22.189648 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.189562    2554 scope.go:117] "RemoveContainer" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.189889 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:22.189858    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": container with ID starting with 742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c not found: ID does not exist" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.189941 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.189900    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"} err="failed to get container status \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": rpc error: code = NotFound desc = could not find container \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": container with ID starting with 742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c not found: ID does not exist"
Jul 18 00:43:22.189941 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.189920    2554 scope.go:117] "RemoveContainer" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.190167 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:22.190152    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": container with ID starting with 91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e not found: ID does not exist" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.190208 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190170    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"} err="failed to get container status \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": rpc error: code = NotFound desc = could not find container \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": container with ID starting with 91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e not found: ID does not exist"
Jul 18 00:43:22.190208 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190183    2554 scope.go:117] "RemoveContainer" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.190408 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:22.190394    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": container with ID starting with 4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011 not found: ID does not exist" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.190454 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190411    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"} err="failed to get container status \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": rpc error: code = NotFound desc = could not find container \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": container with ID starting with 4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011 not found: ID does not exist"
Jul 18 00:43:22.190454 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190427    2554 scope.go:117] "RemoveContainer" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.190720 ip-10-0-141-161 kubenswrapper[2554]: E0718 00:43:22.190695    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": container with ID starting with 80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6 not found: ID does not exist" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.190759 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190729    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"} err="failed to get container status \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": rpc error: code = NotFound desc = could not find container \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": container with ID starting with 80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6 not found: ID does not exist"
Jul 18 00:43:22.190759 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190748    2554 scope.go:117] "RemoveContainer" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.190851 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190837    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:43:22.191004 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.190986    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"} err="failed to get container status \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": rpc error: code = NotFound desc = could not find container \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": container with ID starting with 460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f not found: ID does not exist"
Jul 18 00:43:22.191049 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191005    2554 scope.go:117] "RemoveContainer" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.191179 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191165    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-thanos"
Jul 18 00:43:22.191222 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191184    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-thanos"
Jul 18 00:43:22.191222 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191197    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy"
Jul 18 00:43:22.191222 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191203    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy"
Jul 18 00:43:22.191222 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191205    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"} err="failed to get container status \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": rpc error: code = NotFound desc = could not find container \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": container with ID starting with 141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362 not found: ID does not exist"
Jul 18 00:43:22.191222 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191216    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="thanos-sidecar"
Jul 18 00:43:22.191222 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191222    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="thanos-sidecar"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191233    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-web"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191239    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-web"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191245    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="prometheus"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191250    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="prometheus"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191256    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="config-reloader"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191261    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="config-reloader"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191279    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="init-config-reloader"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191286    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="init-config-reloader"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191221    2554 scope.go:117] "RemoveContainer" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191330    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-web"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191339    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="config-reloader"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191346    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="prometheus"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191354    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191363    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="kube-rbac-proxy-thanos"
Jul 18 00:43:22.191444 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191369    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" containerName="thanos-sidecar"
Jul 18 00:43:22.191999 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191515    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"} err="failed to get container status \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": rpc error: code = NotFound desc = could not find container \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": container with ID starting with 91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30 not found: ID does not exist"
Jul 18 00:43:22.191999 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191536    2554 scope.go:117] "RemoveContainer" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.191999 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191760    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"} err="failed to get container status \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": rpc error: code = NotFound desc = could not find container \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": container with ID starting with 742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c not found: ID does not exist"
Jul 18 00:43:22.191999 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191776    2554 scope.go:117] "RemoveContainer" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.191999 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191985    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"} err="failed to get container status \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": rpc error: code = NotFound desc = could not find container \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": container with ID starting with 91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e not found: ID does not exist"
Jul 18 00:43:22.191999 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.191998    2554 scope.go:117] "RemoveContainer" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.192237 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192216    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"} err="failed to get container status \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": rpc error: code = NotFound desc = could not find container \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": container with ID starting with 4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011 not found: ID does not exist"
Jul 18 00:43:22.192273 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192240    2554 scope.go:117] "RemoveContainer" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.192476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192454    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"} err="failed to get container status \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": rpc error: code = NotFound desc = could not find container \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": container with ID starting with 80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6 not found: ID does not exist"
Jul 18 00:43:22.192476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192476    2554 scope.go:117] "RemoveContainer" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.192704 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192687    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"} err="failed to get container status \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": rpc error: code = NotFound desc = could not find container \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": container with ID starting with 460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f not found: ID does not exist"
Jul 18 00:43:22.192704 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192704    2554 scope.go:117] "RemoveContainer" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.192896 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192877    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"} err="failed to get container status \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": rpc error: code = NotFound desc = could not find container \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": container with ID starting with 141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362 not found: ID does not exist"
Jul 18 00:43:22.192952 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.192896    2554 scope.go:117] "RemoveContainer" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.193094 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193077    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"} err="failed to get container status \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": rpc error: code = NotFound desc = could not find container \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": container with ID starting with 91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30 not found: ID does not exist"
Jul 18 00:43:22.193141 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193095    2554 scope.go:117] "RemoveContainer" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.193275 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193260    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"} err="failed to get container status \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": rpc error: code = NotFound desc = could not find container \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": container with ID starting with 742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c not found: ID does not exist"
Jul 18 00:43:22.193275 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193274    2554 scope.go:117] "RemoveContainer" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.193478 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193461    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"} err="failed to get container status \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": rpc error: code = NotFound desc = could not find container \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": container with ID starting with 91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e not found: ID does not exist"
Jul 18 00:43:22.193478 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193476    2554 scope.go:117] "RemoveContainer" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.193714 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193684    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"} err="failed to get container status \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": rpc error: code = NotFound desc = could not find container \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": container with ID starting with 4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011 not found: ID does not exist"
Jul 18 00:43:22.193714 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193710    2554 scope.go:117] "RemoveContainer" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.193961 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193943    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"} err="failed to get container status \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": rpc error: code = NotFound desc = could not find container \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": container with ID starting with 80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6 not found: ID does not exist"
Jul 18 00:43:22.194001 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.193963    2554 scope.go:117] "RemoveContainer" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.194190 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.194175    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"} err="failed to get container status \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": rpc error: code = NotFound desc = could not find container \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": container with ID starting with 460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f not found: ID does not exist"
Jul 18 00:43:22.194229 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.194191    2554 scope.go:117] "RemoveContainer" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.194402 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.194383    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"} err="failed to get container status \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": rpc error: code = NotFound desc = could not find container \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": container with ID starting with 141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362 not found: ID does not exist"
Jul 18 00:43:22.194402 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.194401    2554 scope.go:117] "RemoveContainer" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.194740 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.194676    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"} err="failed to get container status \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": rpc error: code = NotFound desc = could not find container \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": container with ID starting with 91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30 not found: ID does not exist"
Jul 18 00:43:22.194740 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.194707    2554 scope.go:117] "RemoveContainer" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.195008 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.194985    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"} err="failed to get container status \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": rpc error: code = NotFound desc = could not find container \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": container with ID starting with 742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c not found: ID does not exist"
Jul 18 00:43:22.195008 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.195008    2554 scope.go:117] "RemoveContainer" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.195272 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.195247    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"} err="failed to get container status \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": rpc error: code = NotFound desc = could not find container \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": container with ID starting with 91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e not found: ID does not exist"
Jul 18 00:43:22.195351 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.195274    2554 scope.go:117] "RemoveContainer" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.195518 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.195500    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"} err="failed to get container status \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": rpc error: code = NotFound desc = could not find container \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": container with ID starting with 4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011 not found: ID does not exist"
Jul 18 00:43:22.195569 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.195519    2554 scope.go:117] "RemoveContainer" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.195800 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.195763    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"} err="failed to get container status \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": rpc error: code = NotFound desc = could not find container \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": container with ID starting with 80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6 not found: ID does not exist"
Jul 18 00:43:22.195800 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.195781    2554 scope.go:117] "RemoveContainer" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.196048 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196027    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"} err="failed to get container status \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": rpc error: code = NotFound desc = could not find container \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": container with ID starting with 460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f not found: ID does not exist"
Jul 18 00:43:22.196048 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196043    2554 scope.go:117] "RemoveContainer" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.196267 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196240    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"} err="failed to get container status \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": rpc error: code = NotFound desc = could not find container \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": container with ID starting with 141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362 not found: ID does not exist"
Jul 18 00:43:22.196267 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196259    2554 scope.go:117] "RemoveContainer" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.196476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196452    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"} err="failed to get container status \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": rpc error: code = NotFound desc = could not find container \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": container with ID starting with 91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30 not found: ID does not exist"
Jul 18 00:43:22.196476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196467    2554 scope.go:117] "RemoveContainer" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.196684 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196664    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"} err="failed to get container status \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": rpc error: code = NotFound desc = could not find container \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": container with ID starting with 742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c not found: ID does not exist"
Jul 18 00:43:22.196684 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196682    2554 scope.go:117] "RemoveContainer" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.196913 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196894    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"} err="failed to get container status \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": rpc error: code = NotFound desc = could not find container \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": container with ID starting with 91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e not found: ID does not exist"
Jul 18 00:43:22.196984 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196915    2554 scope.go:117] "RemoveContainer" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.197038 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.196988    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.197224 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.197198    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"} err="failed to get container status \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": rpc error: code = NotFound desc = could not find container \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": container with ID starting with 4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011 not found: ID does not exist"
Jul 18 00:43:22.197356 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.197341    2554 scope.go:117] "RemoveContainer" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.197713 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.197685    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"} err="failed to get container status \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": rpc error: code = NotFound desc = could not find container \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": container with ID starting with 80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6 not found: ID does not exist"
Jul 18 00:43:22.197713 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.197712    2554 scope.go:117] "RemoveContainer" containerID="460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"
Jul 18 00:43:22.197967 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.197946    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f"} err="failed to get container status \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": rpc error: code = NotFound desc = could not find container \"460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f\": container with ID starting with 460ab03b2fb93d2e72da18b8632a821bd076ce204f00c99833ea09621f758a5f not found: ID does not exist"
Jul 18 00:43:22.198044 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.197968    2554 scope.go:117] "RemoveContainer" containerID="141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"
Jul 18 00:43:22.198237 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.198217    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362"} err="failed to get container status \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": rpc error: code = NotFound desc = could not find container \"141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362\": container with ID starting with 141979fcd611e97796558f9449c718f219ca31099ad004290bf2aa5f8c8aa362 not found: ID does not exist"
Jul 18 00:43:22.198287 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.198238    2554 scope.go:117] "RemoveContainer" containerID="91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"
Jul 18 00:43:22.198440 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.198422    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30"} err="failed to get container status \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": rpc error: code = NotFound desc = could not find container \"91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30\": container with ID starting with 91bd0ebd0bfb0fb415d83c8b5b6511e2a08f815e98836266c1f9838b3061bc30 not found: ID does not exist"
Jul 18 00:43:22.198440 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.198439    2554 scope.go:117] "RemoveContainer" containerID="742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"
Jul 18 00:43:22.198680 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.198660    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c"} err="failed to get container status \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": rpc error: code = NotFound desc = could not find container \"742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c\": container with ID starting with 742ce371609c777ed335a13c62cb8dae7dd8e3c69f478c14d24b3cbb9ac4857c not found: ID does not exist"
Jul 18 00:43:22.198680 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.198679    2554 scope.go:117] "RemoveContainer" containerID="91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"
Jul 18 00:43:22.199070 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199046    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e"} err="failed to get container status \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": rpc error: code = NotFound desc = could not find container \"91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e\": container with ID starting with 91006338434d3dda8f7de5a77705fa7f3fe89460b815be853ec5b6ad778eec2e not found: ID does not exist"
Jul 18 00:43:22.199070 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199070    2554 scope.go:117] "RemoveContainer" containerID="4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"
Jul 18 00:43:22.199329 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199303    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011"} err="failed to get container status \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": rpc error: code = NotFound desc = could not find container \"4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011\": container with ID starting with 4ef448fd1282953701e1418c5222c54898c3ec85ec217479d8b9b0c87ecef011 not found: ID does not exist"
Jul 18 00:43:22.199415 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199330    2554 scope.go:117] "RemoveContainer" containerID="80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"
Jul 18 00:43:22.199605 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199583    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6"} err="failed to get container status \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": rpc error: code = NotFound desc = could not find container \"80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6\": container with ID starting with 80363064c76151d5b5de79702cef3ed519d5e29e80df3d197131a02128f3dda6 not found: ID does not exist"
Jul 18 00:43:22.199833 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199818    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 18 00:43:22.199920 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199832    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6fwzg\""
Jul 18 00:43:22.199920 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199832    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-64cvpcic80kml\""
Jul 18 00:43:22.200017 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199957    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 18 00:43:22.200017 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.199968    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 18 00:43:22.200206 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.200191    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 18 00:43:22.200417 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.200404    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 18 00:43:22.200685 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.200670    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 18 00:43:22.201031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.201015    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 18 00:43:22.201214 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.201195    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 18 00:43:22.201214 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.201213    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 18 00:43:22.201352 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.201259    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 18 00:43:22.201352 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.201288    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 18 00:43:22.201512 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.201498    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 18 00:43:22.203608 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.203586    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 18 00:43:22.204691 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.204672    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 18 00:43:22.209904 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.209885    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:43:22.276007 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.275969    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276007 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276006    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276221 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276038    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-config\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276221 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276088    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276221 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276181    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6b714e1-8f8d-482a-93f4-71488ffd7269-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276221 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276206    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276361 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276302    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276397 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276368    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6b714e1-8f8d-482a-93f4-71488ffd7269-config-out\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276457 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276425    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276501 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276486    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276545 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276520    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276593 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276541    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-web-config\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276593 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276572    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276593 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276591    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276801 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276783    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276864 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276837    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276864 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276856    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.276946 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.276884    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.277031 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.277015    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.277078 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.277039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6f8\" (UniqueName: \"kubernetes.io/projected/e6b714e1-8f8d-482a-93f4-71488ffd7269-kube-api-access-lg6f8\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.335983 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.335953    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ab45a4e-eaac-475a-95a0-72cc098b3d6d" path="/var/lib/kubelet/pods/2ab45a4e-eaac-475a-95a0-72cc098b3d6d/volumes"
Jul 18 00:43:22.377933 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.377892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.377933 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.377935    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.377960    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.377981    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.377997    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6f8\" (UniqueName: \"kubernetes.io/projected/e6b714e1-8f8d-482a-93f4-71488ffd7269-kube-api-access-lg6f8\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378026    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378043    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378060    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-config\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378077    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378110    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6b714e1-8f8d-482a-93f4-71488ffd7269-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378138 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378140    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378181    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6b714e1-8f8d-482a-93f4-71488ffd7269-config-out\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378230    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378250    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378274    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378298    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-web-config\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378351    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.378563 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.379019 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.378862    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.379695 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.379666    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.380847 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.380823    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6b714e1-8f8d-482a-93f4-71488ffd7269-config-out\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.381193 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.381172    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.381283 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.381251    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.381346 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.381316    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.381346 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.381328    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.381454 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.381389    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.382242 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.381936    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.382242 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.381993    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.382242 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.382115    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6b714e1-8f8d-482a-93f4-71488ffd7269-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.382451 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.382379    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.382664 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.382639    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.382943 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.382921    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.383227 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.383209    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6b714e1-8f8d-482a-93f4-71488ffd7269-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.383504 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.383481    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-config\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.383600 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.383582    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.383884 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.383866    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-web-config\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.384217 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.384201    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6b714e1-8f8d-482a-93f4-71488ffd7269-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.386119 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.386104    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6f8\" (UniqueName: \"kubernetes.io/projected/e6b714e1-8f8d-482a-93f4-71488ffd7269-kube-api-access-lg6f8\") pod \"prometheus-k8s-0\" (UID: \"e6b714e1-8f8d-482a-93f4-71488ffd7269\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.508588 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.508559    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:43:22.640554 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:22.640487    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 00:43:22.643167 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:43:22.643139    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b714e1_8f8d_482a_93f4_71488ffd7269.slice/crio-1da9cb45a843ee9e2396577aa415279424dd49ace64c772f48ca2bfa75745454 WatchSource:0}: Error finding container 1da9cb45a843ee9e2396577aa415279424dd49ace64c772f48ca2bfa75745454: Status 404 returned error can't find the container with id 1da9cb45a843ee9e2396577aa415279424dd49ace64c772f48ca2bfa75745454
Jul 18 00:43:23.144345 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:23.144307    2554 generic.go:356] "Generic (PLEG): container finished" podID="e6b714e1-8f8d-482a-93f4-71488ffd7269" containerID="660367f784b5460b71875d64b706ceabd377149f2be0dc19dc1c5f21cc15d8ec" exitCode=0
Jul 18 00:43:23.144791 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:23.144396    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerDied","Data":"660367f784b5460b71875d64b706ceabd377149f2be0dc19dc1c5f21cc15d8ec"}
Jul 18 00:43:23.144791 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:23.144494    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerStarted","Data":"1da9cb45a843ee9e2396577aa415279424dd49ace64c772f48ca2bfa75745454"}
Jul 18 00:43:24.151716 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:24.151676    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerStarted","Data":"7be34dc676d6332b57f5f01399e03b6a1a5327860233a6cf375c7e6eeb40d801"}
Jul 18 00:43:24.152101 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:24.151751    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerStarted","Data":"e14840f4a3b2ac1f24218dd60dada18f2b2a8adce05a6c2cee39cfeadf521d00"}
Jul 18 00:43:24.152101 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:24.151777    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerStarted","Data":"a5d8239f41d100c0aa3b1b272ba3c79a0f87256a937616bfc74f93f09edfc575"}
Jul 18 00:43:24.152101 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:24.151799    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerStarted","Data":"d226294780776318800517b38340c0aa9481eae75ca70bba032f7ec76578fa27"}
Jul 18 00:43:24.152101 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:24.151823    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerStarted","Data":"80353275a20a582e85c2d4543e4001f3a891c1be3015766750be155338057a2c"}
Jul 18 00:43:24.152101 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:24.151845    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6b714e1-8f8d-482a-93f4-71488ffd7269","Type":"ContainerStarted","Data":"294a93ae8ab475caf3b8fffd3757653bbc237362f000637cdc441e11b7bad80b"}
Jul 18 00:43:24.180103 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:24.180051    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.161798093 podStartE2EDuration="2.18003682s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.238727ms" isStatefulPod=false podCreationTimestamp="2026-07-18 00:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:43:24.178077005 +0000 UTC m=+160.304011985" watchObservedRunningTime="2026-07-18 00:43:24.18003682 +0000 UTC m=+160.305971798"
Jul 18 00:43:27.508877 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:43:27.508837    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:44:22.508895 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:44:22.508860    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:44:22.524332 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:44:22.524309    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:44:23.387476 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:44:23.387447    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 00:45:44.216472 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:44.216446    2554 kubelet.go:1648] "Image garbage collection succeeded"
Jul 18 00:45:54.296193 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.296155    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6n2wn"]
Jul 18 00:45:54.299190 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.299165    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.302339 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.302311    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 18 00:45:54.302453 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.302342    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jrjp8\""
Jul 18 00:45:54.302453 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.302342    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 18 00:45:54.309957 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.309930    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6n2wn"]
Jul 18 00:45:54.488092 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.488058    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzb6\" (UniqueName: \"kubernetes.io/projected/5f5a9526-e830-4ab2-b7cf-12d79c5da172-kube-api-access-5tzb6\") pod \"cert-manager-webhook-6486ffd997-6n2wn\" (UID: \"5f5a9526-e830-4ab2-b7cf-12d79c5da172\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.488309 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.488119    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f5a9526-e830-4ab2-b7cf-12d79c5da172-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6n2wn\" (UID: \"5f5a9526-e830-4ab2-b7cf-12d79c5da172\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.588738 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.588646    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzb6\" (UniqueName: \"kubernetes.io/projected/5f5a9526-e830-4ab2-b7cf-12d79c5da172-kube-api-access-5tzb6\") pod \"cert-manager-webhook-6486ffd997-6n2wn\" (UID: \"5f5a9526-e830-4ab2-b7cf-12d79c5da172\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.588738 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.588707    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f5a9526-e830-4ab2-b7cf-12d79c5da172-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6n2wn\" (UID: \"5f5a9526-e830-4ab2-b7cf-12d79c5da172\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.596597 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.596573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f5a9526-e830-4ab2-b7cf-12d79c5da172-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6n2wn\" (UID: \"5f5a9526-e830-4ab2-b7cf-12d79c5da172\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.596738 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.596645    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzb6\" (UniqueName: \"kubernetes.io/projected/5f5a9526-e830-4ab2-b7cf-12d79c5da172-kube-api-access-5tzb6\") pod \"cert-manager-webhook-6486ffd997-6n2wn\" (UID: \"5f5a9526-e830-4ab2-b7cf-12d79c5da172\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.623227 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.623198    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:54.754380 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.754357    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6n2wn"]
Jul 18 00:45:54.756453 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:45:54.756424    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f5a9526_e830_4ab2_b7cf_12d79c5da172.slice/crio-cb0e95400bcb21976c6cf070706f5343e06e7b2bd95d5c820bfed14ed48693f5 WatchSource:0}: Error finding container cb0e95400bcb21976c6cf070706f5343e06e7b2bd95d5c820bfed14ed48693f5: Status 404 returned error can't find the container with id cb0e95400bcb21976c6cf070706f5343e06e7b2bd95d5c820bfed14ed48693f5
Jul 18 00:45:54.758289 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:54.758271    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 00:45:55.735760 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:55.735711    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn" event={"ID":"5f5a9526-e830-4ab2-b7cf-12d79c5da172","Type":"ContainerStarted","Data":"cb0e95400bcb21976c6cf070706f5343e06e7b2bd95d5c820bfed14ed48693f5"}
Jul 18 00:45:58.749424 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:58.749386    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn" event={"ID":"5f5a9526-e830-4ab2-b7cf-12d79c5da172","Type":"ContainerStarted","Data":"f7a6bd2d3ad045b30a5f0ba2d20cefbfd54c7fe439147f739aad978d94da5e82"}
Jul 18 00:45:58.749848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:58.749446    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:45:58.767560 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:45:58.767501    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn" podStartSLOduration=1.756678787 podStartE2EDuration="4.767486387s" totalImagesPullingTime="3.0108076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:45:58.766127332 +0000 UTC m=+314.892062310" watchObservedRunningTime="2026-07-18 00:45:58.767486387 +0000 UTC m=+314.893421365"
Jul 18 00:46:03.743779 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.743743    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nf9mb"]
Jul 18 00:46:03.746936 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.746918    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:03.749536 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.749512    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-r5cw7\""
Jul 18 00:46:03.757753 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.757731    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nf9mb"]
Jul 18 00:46:03.765352 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.765330    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af5a80c-4811-4b21-a100-589b29e0d01e-bound-sa-token\") pod \"cert-manager-55b4d954d5-nf9mb\" (UID: \"3af5a80c-4811-4b21-a100-589b29e0d01e\") " pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:03.765468 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.765375    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7r77\" (UniqueName: \"kubernetes.io/projected/3af5a80c-4811-4b21-a100-589b29e0d01e-kube-api-access-f7r77\") pod \"cert-manager-55b4d954d5-nf9mb\" (UID: \"3af5a80c-4811-4b21-a100-589b29e0d01e\") " pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:03.866849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.866790    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af5a80c-4811-4b21-a100-589b29e0d01e-bound-sa-token\") pod \"cert-manager-55b4d954d5-nf9mb\" (UID: \"3af5a80c-4811-4b21-a100-589b29e0d01e\") " pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:03.866849 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.866867    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7r77\" (UniqueName: \"kubernetes.io/projected/3af5a80c-4811-4b21-a100-589b29e0d01e-kube-api-access-f7r77\") pod \"cert-manager-55b4d954d5-nf9mb\" (UID: \"3af5a80c-4811-4b21-a100-589b29e0d01e\") " pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:03.875148 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.875121    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af5a80c-4811-4b21-a100-589b29e0d01e-bound-sa-token\") pod \"cert-manager-55b4d954d5-nf9mb\" (UID: \"3af5a80c-4811-4b21-a100-589b29e0d01e\") " pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:03.875264 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:03.875163    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7r77\" (UniqueName: \"kubernetes.io/projected/3af5a80c-4811-4b21-a100-589b29e0d01e-kube-api-access-f7r77\") pod \"cert-manager-55b4d954d5-nf9mb\" (UID: \"3af5a80c-4811-4b21-a100-589b29e0d01e\") " pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:04.055779 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:04.055696    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nf9mb"
Jul 18 00:46:04.184378 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:04.184348    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nf9mb"]
Jul 18 00:46:04.187779 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:46:04.187740    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3af5a80c_4811_4b21_a100_589b29e0d01e.slice/crio-55997041a30c532a75bdb656bf692f9d87862ecbce55b736c6bb3f41881eccb9 WatchSource:0}: Error finding container 55997041a30c532a75bdb656bf692f9d87862ecbce55b736c6bb3f41881eccb9: Status 404 returned error can't find the container with id 55997041a30c532a75bdb656bf692f9d87862ecbce55b736c6bb3f41881eccb9
Jul 18 00:46:04.755113 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:04.755081    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6n2wn"
Jul 18 00:46:04.775596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:04.775559    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nf9mb" event={"ID":"3af5a80c-4811-4b21-a100-589b29e0d01e","Type":"ContainerStarted","Data":"17d74d8950f4aa343e040d1ab2b14facc5dc939a43c42fb4a32e2b1bbef49d76"}
Jul 18 00:46:04.775596 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:04.775599    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nf9mb" event={"ID":"3af5a80c-4811-4b21-a100-589b29e0d01e","Type":"ContainerStarted","Data":"55997041a30c532a75bdb656bf692f9d87862ecbce55b736c6bb3f41881eccb9"}
Jul 18 00:46:04.790964 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:04.790911    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nf9mb" podStartSLOduration=1.790892833 podStartE2EDuration="1.790892833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:04.790111578 +0000 UTC m=+320.916046557" watchObservedRunningTime="2026-07-18 00:46:04.790892833 +0000 UTC m=+320.916827811"
Jul 18 00:46:29.357905 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.357867    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"]
Jul 18 00:46:29.362280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.362256    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.367381 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.367355    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 18 00:46:29.367555 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.367536    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 18 00:46:29.367652 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.367538    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 18 00:46:29.367717 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.367681    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 18 00:46:29.368461 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.368441    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-46br2\""
Jul 18 00:46:29.396813 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.396781    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"]
Jul 18 00:46:29.486796 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.486757    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-manifests\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.486991 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.486798    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.486991 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.486886    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.486991 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.486941    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-tmp\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.486991 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.486981    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnr4r\" (UniqueName: \"kubernetes.io/projected/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-kube-api-access-tnr4r\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.587981 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.587939    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnr4r\" (UniqueName: \"kubernetes.io/projected/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-kube-api-access-tnr4r\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.588197 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.588013    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-manifests\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.588197 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.588030    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.588197 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.588058    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.588197 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.588089    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-tmp\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.588413 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.588393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-manifests\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.590383 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.590352    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-tmp\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.590509 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.590487    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.590708 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.590689    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.595836 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.595816    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnr4r\" (UniqueName: \"kubernetes.io/projected/fa3e6edb-9b8c-4b31-a1b6-d234dee87f72-kube-api-access-tnr4r\") pod \"opendatahub-operator-controller-manager-85c6dd979f-mm688\" (UID: \"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.674941 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.674868    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:29.806170 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.806144    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"]
Jul 18 00:46:29.808570 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:46:29.808528    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3e6edb_9b8c_4b31_a1b6_d234dee87f72.slice/crio-84dbf0a236fa2c2906cdf44edb98dff020a2a9705c04bb130366577ad153be9f WatchSource:0}: Error finding container 84dbf0a236fa2c2906cdf44edb98dff020a2a9705c04bb130366577ad153be9f: Status 404 returned error can't find the container with id 84dbf0a236fa2c2906cdf44edb98dff020a2a9705c04bb130366577ad153be9f
Jul 18 00:46:29.879029 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:29.878993    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688" event={"ID":"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72","Type":"ContainerStarted","Data":"84dbf0a236fa2c2906cdf44edb98dff020a2a9705c04bb130366577ad153be9f"}
Jul 18 00:46:33.899022 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:33.898985    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688" event={"ID":"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72","Type":"ContainerStarted","Data":"a7203d15a378710d853d33a92a7f71383bc53d4a0e8736debe40fdc5bac4c2b6"}
Jul 18 00:46:34.905186 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:34.905153    2554 generic.go:356] "Generic (PLEG): container finished" podID="fa3e6edb-9b8c-4b31-a1b6-d234dee87f72" containerID="a7203d15a378710d853d33a92a7f71383bc53d4a0e8736debe40fdc5bac4c2b6" exitCode=0
Jul 18 00:46:34.905582 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:34.905235    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688" event={"ID":"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72","Type":"ContainerDied","Data":"a7203d15a378710d853d33a92a7f71383bc53d4a0e8736debe40fdc5bac4c2b6"}
Jul 18 00:46:35.912221 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:35.912189    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688" event={"ID":"fa3e6edb-9b8c-4b31-a1b6-d234dee87f72","Type":"ContainerStarted","Data":"a75647b8f98ab08aa3cd66e4a739a44eace7eb7b26d8f1b9decc64e870f21306"}
Jul 18 00:46:35.912661 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:35.912299    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:35.935202 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:35.935154    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688" podStartSLOduration=2.430125556 podStartE2EDuration="6.93513863s" totalImagesPullingTime="3.412335586s" totalInitContainerRuntime="1.092677488s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:35.93440096 +0000 UTC m=+352.060335938" watchObservedRunningTime="2026-07-18 00:46:35.93513863 +0000 UTC m=+352.061073609"
Jul 18 00:46:46.919099 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:46.919068    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-mm688"
Jul 18 00:46:56.210179 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.210148    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"]
Jul 18 00:46:56.218848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.218823    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.221455 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.221429    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k5hfz\""
Jul 18 00:46:56.221583 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.221459    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 00:46:56.221583 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.221554    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 18 00:46:56.221694 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.221594    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 18 00:46:56.225848 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.225825    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"]
Jul 18 00:46:56.322084 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322042    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322084 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322088    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spt2\" (UniqueName: \"kubernetes.io/projected/593c9c0e-8705-409e-a2d4-0546e31fe99d-kube-api-access-6spt2\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322351 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322125    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322351 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322227    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322351 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322256    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322351 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322295    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322556 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322406    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322556 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322457    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c9c0e-8705-409e-a2d4-0546e31fe99d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.322556 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.322544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.423741 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.423741 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423744    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424002 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423764    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spt2\" (UniqueName: \"kubernetes.io/projected/593c9c0e-8705-409e-a2d4-0546e31fe99d-kube-api-access-6spt2\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424002 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423783    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424002 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423818    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424002 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423850    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424002 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423891    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424002 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423936    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424002 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.423964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c9c0e-8705-409e-a2d4-0546e31fe99d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424356 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.424202    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424356 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.424247    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424464 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.424433    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424519 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.424501    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.424835 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.424813    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c9c0e-8705-409e-a2d4-0546e31fe99d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.426470 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.426446    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.426642 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.426607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.436669 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.436643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593c9c0e-8705-409e-a2d4-0546e31fe99d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.436785 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.436676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spt2\" (UniqueName: \"kubernetes.io/projected/593c9c0e-8705-409e-a2d4-0546e31fe99d-kube-api-access-6spt2\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hf2d6\" (UID: \"593c9c0e-8705-409e-a2d4-0546e31fe99d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.530281 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.530246    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:46:56.690969 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:56.690940    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"]
Jul 18 00:46:56.693649 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:46:56.693581    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593c9c0e_8705_409e_a2d4_0546e31fe99d.slice/crio-efd8d1ae6c893530806775eff3a103216696bd5444d6fcf6744ed96522c34195 WatchSource:0}: Error finding container efd8d1ae6c893530806775eff3a103216696bd5444d6fcf6744ed96522c34195: Status 404 returned error can't find the container with id efd8d1ae6c893530806775eff3a103216696bd5444d6fcf6744ed96522c34195
Jul 18 00:46:57.011428 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.011399    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6" event={"ID":"593c9c0e-8705-409e-a2d4-0546e31fe99d","Type":"ContainerStarted","Data":"efd8d1ae6c893530806775eff3a103216696bd5444d6fcf6744ed96522c34195"}
Jul 18 00:46:57.956055 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.956013    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"]
Jul 18 00:46:57.959677 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.959655    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:57.962288 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.962266    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 18 00:46:57.963037 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.963012    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 18 00:46:57.964247 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.964226    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 18 00:46:57.964451 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.964369    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 18 00:46:57.964451 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.964440    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 18 00:46:57.964635 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.964444    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5nglk\""
Jul 18 00:46:57.976817 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:57.976789    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"]
Jul 18 00:46:58.037850 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.037822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5644a48d-2976-4b6f-b410-384b1bf72056-manager-config\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.038089 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.037906    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjds\" (UniqueName: \"kubernetes.io/projected/5644a48d-2976-4b6f-b410-384b1bf72056-kube-api-access-kvjds\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.038089 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.037945    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5644a48d-2976-4b6f-b410-384b1bf72056-cert\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.038089 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.037972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5644a48d-2976-4b6f-b410-384b1bf72056-metrics-cert\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.139357 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.139312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjds\" (UniqueName: \"kubernetes.io/projected/5644a48d-2976-4b6f-b410-384b1bf72056-kube-api-access-kvjds\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.139357 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.139364    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5644a48d-2976-4b6f-b410-384b1bf72056-cert\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.139641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.139390    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5644a48d-2976-4b6f-b410-384b1bf72056-metrics-cert\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.139641 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.139463    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5644a48d-2976-4b6f-b410-384b1bf72056-manager-config\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.140314 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.140285    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5644a48d-2976-4b6f-b410-384b1bf72056-manager-config\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.142661 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.142636    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5644a48d-2976-4b6f-b410-384b1bf72056-metrics-cert\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.142661 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.142647    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5644a48d-2976-4b6f-b410-384b1bf72056-cert\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.152953 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.152929    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjds\" (UniqueName: \"kubernetes.io/projected/5644a48d-2976-4b6f-b410-384b1bf72056-kube-api-access-kvjds\") pod \"lws-controller-manager-5b88d5d556-td8hq\" (UID: \"5644a48d-2976-4b6f-b410-384b1bf72056\") " pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.271445 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.271406    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:46:58.426099 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.425989    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"]
Jul 18 00:46:58.944789 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.944750    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"]
Jul 18 00:46:58.952100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.952049    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:58.955489 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.955460    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 18 00:46:58.955646 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.955528    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 18 00:46:58.955925 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.955856    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 18 00:46:58.956240 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.956222    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6rkdc\""
Jul 18 00:46:58.991524 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:58.991474    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"]
Jul 18 00:46:59.048434 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.048397    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.048661 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.048447    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.048661 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.048495    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.048661 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.048520    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.048822 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.048673    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.048822 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.048732    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlbz\" (UniqueName: \"kubernetes.io/projected/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-kube-api-access-bdlbz\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.048822 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.048763    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.149666 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.149605    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlbz\" (UniqueName: \"kubernetes.io/projected/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-kube-api-access-bdlbz\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.149666 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.149670    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.149886 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.149724    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.149886 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.149749    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.149973 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.149889    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.149973 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.149926    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.150076 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.149994    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.150817 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.150790    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.151993 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.151970    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.152240 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.152217    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.152519 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.152499    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.152561 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.152509    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.166728 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.166705    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlbz\" (UniqueName: \"kubernetes.io/projected/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-kube-api-access-bdlbz\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.169405 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.169383    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1510d1-f338-4b0d-a8d8-969029c8dbc4-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ff5sb\" (UID: \"6d1510d1-f338-4b0d-a8d8-969029c8dbc4\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.264207 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.264167    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:46:59.378457 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:46:59.378412    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5644a48d_2976_4b6f_b410_384b1bf72056.slice/crio-e3a9a01c297d9b89986aa58c093b017a87f16434e13dfeaffb1222e215fb10ca WatchSource:0}: Error finding container e3a9a01c297d9b89986aa58c093b017a87f16434e13dfeaffb1222e215fb10ca: Status 404 returned error can't find the container with id e3a9a01c297d9b89986aa58c093b017a87f16434e13dfeaffb1222e215fb10ca
Jul 18 00:46:59.460971 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.460928    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:46:59.461100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.461021    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:46:59.461100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.461061    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:46:59.555948 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:46:59.555912    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"]
Jul 18 00:46:59.562190 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:46:59.562148    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1510d1_f338_4b0d_a8d8_969029c8dbc4.slice/crio-f25c0e1db32880a0d897a77493a5e25a126e456f9a707e00196a29e309590902 WatchSource:0}: Error finding container f25c0e1db32880a0d897a77493a5e25a126e456f9a707e00196a29e309590902: Status 404 returned error can't find the container with id f25c0e1db32880a0d897a77493a5e25a126e456f9a707e00196a29e309590902
Jul 18 00:47:00.028115 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:00.028074    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb" event={"ID":"6d1510d1-f338-4b0d-a8d8-969029c8dbc4","Type":"ContainerStarted","Data":"f25c0e1db32880a0d897a77493a5e25a126e456f9a707e00196a29e309590902"}
Jul 18 00:47:00.029879 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:00.029843    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6" event={"ID":"593c9c0e-8705-409e-a2d4-0546e31fe99d","Type":"ContainerStarted","Data":"771188185bd036e194bf4fe6e3bc899508ef79e80d4bfbe00341b016b5eff2e9"}
Jul 18 00:47:00.031064 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:00.031039    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq" event={"ID":"5644a48d-2976-4b6f-b410-384b1bf72056","Type":"ContainerStarted","Data":"e3a9a01c297d9b89986aa58c093b017a87f16434e13dfeaffb1222e215fb10ca"}
Jul 18 00:47:00.068829 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:00.068764    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6" podStartSLOduration=1.303867207 podStartE2EDuration="4.068740518s" totalImagesPullingTime="2.764873311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:00.066044423 +0000 UTC m=+376.191979401" watchObservedRunningTime="2026-07-18 00:47:00.068740518 +0000 UTC m=+376.194675499"
Jul 18 00:47:00.530922 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:00.530885    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:47:00.537859 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:00.537635    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:47:01.036675 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:01.036642    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:47:01.037920 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:01.037881    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hf2d6"
Jul 18 00:47:02.876920 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:02.876883    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:47:02.877251 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:02.876967    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:47:03.047033 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.046996    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq" event={"ID":"5644a48d-2976-4b6f-b410-384b1bf72056","Type":"ContainerStarted","Data":"9180dd516c0ddae5358474594c36a8dc0c4458bf31e1907c5eabc6aa79b4935c"}
Jul 18 00:47:03.047221 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.047149    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:47:03.048708 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.048683    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb" event={"ID":"6d1510d1-f338-4b0d-a8d8-969029c8dbc4","Type":"ContainerStarted","Data":"76e00385e4460b9493185ffadbdadfc1f20cbed00b626f2dc755e1ada15bed95"}
Jul 18 00:47:03.048810 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.048724    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:47:03.050439 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.050414    2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-ff5sb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 00:47:03.050530 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.050462    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb" podUID="6d1510d1-f338-4b0d-a8d8-969029c8dbc4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 00:47:03.066268 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.066209    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq" podStartSLOduration=2.572276526 podStartE2EDuration="6.066192968s" totalImagesPullingTime="3.493916442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:03.065731125 +0000 UTC m=+379.191666105" watchObservedRunningTime="2026-07-18 00:47:03.066192968 +0000 UTC m=+379.192127946"
Jul 18 00:47:03.086585 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:03.086311    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb" podStartSLOduration=1.774102633 podStartE2EDuration="5.086289574s" totalImagesPullingTime="3.312186941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:03.085685266 +0000 UTC m=+379.211620248" watchObservedRunningTime="2026-07-18 00:47:03.086289574 +0000 UTC m=+379.212224557"
Jul 18 00:47:04.054320 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:04.054294    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ff5sb"
Jul 18 00:47:14.055858 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:47:14.055820    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b88d5d556-td8hq"
Jul 18 00:49:33.055771 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.055738    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"]
Jul 18 00:49:33.059384 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.059363    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.062001 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.061980    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wgb5b\""
Jul 18 00:49:33.070231 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.070208    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"]
Jul 18 00:49:33.152917 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.152884    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.152967    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.152990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.153020    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.153043    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153072 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.153066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153234 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.153081    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153234 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.153100    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.153234 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.153134    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfd2q\" (UniqueName: \"kubernetes.io/projected/a8c9e7b2-5600-4833-96ce-79b2fe2df033-kube-api-access-hfd2q\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254170 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254132    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254170 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254165    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254381 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254196    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254381 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254218    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254381 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254242    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254381 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254266    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254381 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254290    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254381 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254311    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfd2q\" (UniqueName: \"kubernetes.io/projected/a8c9e7b2-5600-4833-96ce-79b2fe2df033-kube-api-access-hfd2q\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254759 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254759 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254577    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254759 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254644    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254759 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254725    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.254966 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.254942    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.255088 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.255062    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.256747 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.256726    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.256841 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.256775    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.262733 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.262712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e7b2-5600-4833-96ce-79b2fe2df033-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.262733 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.262732    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfd2q\" (UniqueName: \"kubernetes.io/projected/a8c9e7b2-5600-4833-96ce-79b2fe2df033-kube-api-access-hfd2q\") pod \"maas-default-gateway-openshift-default-56654dcf7-rwfcw\" (UID: \"a8c9e7b2-5600-4833-96ce-79b2fe2df033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.370070 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.370001    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:33.506271 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.506246    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"]
Jul 18 00:49:33.508155 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:49:33.508129    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c9e7b2_5600_4833_96ce_79b2fe2df033.slice/crio-29928fac27350194ec4d02451a894e4779177f7de0fa1a10e959a480ca1cf4aa WatchSource:0}: Error finding container 29928fac27350194ec4d02451a894e4779177f7de0fa1a10e959a480ca1cf4aa: Status 404 returned error can't find the container with id 29928fac27350194ec4d02451a894e4779177f7de0fa1a10e959a480ca1cf4aa
Jul 18 00:49:33.510266 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.510235    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:49:33.510334 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.510302    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:49:33.510371 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.510335    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:49:33.732218 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.732142    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" event={"ID":"a8c9e7b2-5600-4833-96ce-79b2fe2df033","Type":"ContainerStarted","Data":"2d358b5c58b5c7b1c6e68948dc9e2f05fb29f3c8e1fb054422a45a1bea970d5f"}
Jul 18 00:49:33.732218 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.732193    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" event={"ID":"a8c9e7b2-5600-4833-96ce-79b2fe2df033","Type":"ContainerStarted","Data":"29928fac27350194ec4d02451a894e4779177f7de0fa1a10e959a480ca1cf4aa"}
Jul 18 00:49:33.759095 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:33.759049    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" podStartSLOduration=0.759034088 podStartE2EDuration="759.034088ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:33.757962873 +0000 UTC m=+529.883897853" watchObservedRunningTime="2026-07-18 00:49:33.759034088 +0000 UTC m=+529.884969066"
Jul 18 00:49:34.370589 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:34.370559    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:34.375100 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:34.375065    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:34.737043 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:34.736956    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:49:34.737798 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:49:34.737783    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:50:06.548454 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.548419    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"]
Jul 18 00:50:06.552156 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.552125    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:06.561463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.558913    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 18 00:50:06.561463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.558991    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 00:50:06.561463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.560879    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 00:50:06.561463 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.560964    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8w5cv\""
Jul 18 00:50:06.563810 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.563782    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"]
Jul 18 00:50:06.665349 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.665304    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11b679f6-cb84-4f78-a4b9-8b3bdf351301-data\") pod \"postgres-69fb4594cf-whmbf\" (UID: \"11b679f6-cb84-4f78-a4b9-8b3bdf351301\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:06.665535 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.665448    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5648j\" (UniqueName: \"kubernetes.io/projected/11b679f6-cb84-4f78-a4b9-8b3bdf351301-kube-api-access-5648j\") pod \"postgres-69fb4594cf-whmbf\" (UID: \"11b679f6-cb84-4f78-a4b9-8b3bdf351301\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:06.766792 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.766745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5648j\" (UniqueName: \"kubernetes.io/projected/11b679f6-cb84-4f78-a4b9-8b3bdf351301-kube-api-access-5648j\") pod \"postgres-69fb4594cf-whmbf\" (UID: \"11b679f6-cb84-4f78-a4b9-8b3bdf351301\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:06.766991 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.766838    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11b679f6-cb84-4f78-a4b9-8b3bdf351301-data\") pod \"postgres-69fb4594cf-whmbf\" (UID: \"11b679f6-cb84-4f78-a4b9-8b3bdf351301\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:06.767245 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.767223    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11b679f6-cb84-4f78-a4b9-8b3bdf351301-data\") pod \"postgres-69fb4594cf-whmbf\" (UID: \"11b679f6-cb84-4f78-a4b9-8b3bdf351301\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:06.775310 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.775273    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5648j\" (UniqueName: \"kubernetes.io/projected/11b679f6-cb84-4f78-a4b9-8b3bdf351301-kube-api-access-5648j\") pod \"postgres-69fb4594cf-whmbf\" (UID: \"11b679f6-cb84-4f78-a4b9-8b3bdf351301\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:06.870490 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:06.870403    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:07.008154 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:07.008127    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"]
Jul 18 00:50:07.010639 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:50:07.010590    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b679f6_cb84_4f78_a4b9_8b3bdf351301.slice/crio-03b7e25fc3dc88482a7dfd0dff639db54c114a30d7ec42c45c12816c335c8749 WatchSource:0}: Error finding container 03b7e25fc3dc88482a7dfd0dff639db54c114a30d7ec42c45c12816c335c8749: Status 404 returned error can't find the container with id 03b7e25fc3dc88482a7dfd0dff639db54c114a30d7ec42c45c12816c335c8749
Jul 18 00:50:07.889604 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:07.889565    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf" event={"ID":"11b679f6-cb84-4f78-a4b9-8b3bdf351301","Type":"ContainerStarted","Data":"03b7e25fc3dc88482a7dfd0dff639db54c114a30d7ec42c45c12816c335c8749"}
Jul 18 00:50:11.912565 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:11.912517    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf" event={"ID":"11b679f6-cb84-4f78-a4b9-8b3bdf351301","Type":"ContainerStarted","Data":"8fa93f8b5e0ef1c7e54e0f5ec01a87d6b2d856ff8423ac8c3052016ae6a6bbbb"}
Jul 18 00:50:11.913117 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:11.912604    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:50:11.933413 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:11.933345    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf" podStartSLOduration=1.13652492 podStartE2EDuration="5.933321814s" totalImagesPullingTime="4.796796894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:50:11.930828591 +0000 UTC m=+568.056763573" watchObservedRunningTime="2026-07-18 00:50:11.933321814 +0000 UTC m=+568.059256795"
Jul 18 00:50:17.946706 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:50:17.946665    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-whmbf"
Jul 18 00:51:01.062010 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.061973    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"]
Jul 18 00:51:01.065539 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.065518    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.068021 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.068003    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 18 00:51:01.068099 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.068044    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-phsvb\""
Jul 18 00:51:01.074004 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.073981    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"]
Jul 18 00:51:01.083373 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.083341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cqp9\" (UniqueName: \"kubernetes.io/projected/dfb9430f-32f7-40a0-b155-b06cd7c55f37-kube-api-access-8cqp9\") pod \"payload-pre-processing-7677b774bb-zvmsp\" (UID: \"dfb9430f-32f7-40a0-b155-b06cd7c55f37\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.083534 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.083397    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb9430f-32f7-40a0-b155-b06cd7c55f37-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-zvmsp\" (UID: \"dfb9430f-32f7-40a0-b155-b06cd7c55f37\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.184490 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.184452    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cqp9\" (UniqueName: \"kubernetes.io/projected/dfb9430f-32f7-40a0-b155-b06cd7c55f37-kube-api-access-8cqp9\") pod \"payload-pre-processing-7677b774bb-zvmsp\" (UID: \"dfb9430f-32f7-40a0-b155-b06cd7c55f37\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.184708 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.184516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb9430f-32f7-40a0-b155-b06cd7c55f37-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-zvmsp\" (UID: \"dfb9430f-32f7-40a0-b155-b06cd7c55f37\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.185527 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.185504    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb9430f-32f7-40a0-b155-b06cd7c55f37-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-zvmsp\" (UID: \"dfb9430f-32f7-40a0-b155-b06cd7c55f37\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.193333 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.193295    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cqp9\" (UniqueName: \"kubernetes.io/projected/dfb9430f-32f7-40a0-b155-b06cd7c55f37-kube-api-access-8cqp9\") pod \"payload-pre-processing-7677b774bb-zvmsp\" (UID: \"dfb9430f-32f7-40a0-b155-b06cd7c55f37\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.377032 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.376944    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:01.523170 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.523125    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"]
Jul 18 00:51:01.549390 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:01.549335    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 00:51:02.156012 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:02.155966    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp" event={"ID":"dfb9430f-32f7-40a0-b155-b06cd7c55f37","Type":"ContainerStarted","Data":"ad6bce06507772b727651ace184ae1bc1f82a9580362471602a481b3147e6409"}
Jul 18 00:51:04.167028 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:04.166995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp" event={"ID":"dfb9430f-32f7-40a0-b155-b06cd7c55f37","Type":"ContainerStarted","Data":"b4987dab204f2d2f764cc5aaa708bc322b105a1fa55faf4e26dbe138e48b30a5"}
Jul 18 00:51:04.167431 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:04.167075    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:04.184161 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:04.184086    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp" podStartSLOduration=1.5061547100000001 podStartE2EDuration="3.184056627s" totalImagesPullingTime="1.677901917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:04.181072473 +0000 UTC m=+620.307007467" watchObservedRunningTime="2026-07-18 00:51:04.184056627 +0000 UTC m=+620.309991610"
Jul 18 00:51:09.193272 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:09.193236    2554 generic.go:356] "Generic (PLEG): container finished" podID="a8c9e7b2-5600-4833-96ce-79b2fe2df033" containerID="2d358b5c58b5c7b1c6e68948dc9e2f05fb29f3c8e1fb054422a45a1bea970d5f" exitCode=0
Jul 18 00:51:09.193726 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:09.193310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" event={"ID":"a8c9e7b2-5600-4833-96ce-79b2fe2df033","Type":"ContainerDied","Data":"2d358b5c58b5c7b1c6e68948dc9e2f05fb29f3c8e1fb054422a45a1bea970d5f"}
Jul 18 00:51:09.193774 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:09.193741    2554 scope.go:117] "RemoveContainer" containerID="2d358b5c58b5c7b1c6e68948dc9e2f05fb29f3c8e1fb054422a45a1bea970d5f"
Jul 18 00:51:09.194724 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:09.194698    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:51:09.194787 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:09.194771    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:51:09.194822 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:09.194806    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 00:51:10.200412 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:10.200377    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" event={"ID":"a8c9e7b2-5600-4833-96ce-79b2fe2df033","Type":"ContainerStarted","Data":"e71738c6ffa61932416167f88ce973dd5cdd0e55ba427d9855ac06418245c5b2"}
Jul 18 00:51:10.370376 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:10.370336    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:51:10.370693 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:10.370670    2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rwfcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 18 00:51:10.370757 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:10.370723    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" podUID="a8c9e7b2-5600-4833-96ce-79b2fe2df033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 18 00:51:11.371057 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:11.371012    2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rwfcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 18 00:51:11.371573 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:11.371075    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" podUID="a8c9e7b2-5600-4833-96ce-79b2fe2df033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 18 00:51:12.370814 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:12.370775    2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rwfcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 18 00:51:12.371053 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:12.370831    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" podUID="a8c9e7b2-5600-4833-96ce-79b2fe2df033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 18 00:51:13.370850 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:13.370815    2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rwfcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 18 00:51:13.371266 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:13.370872    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw" podUID="a8c9e7b2-5600-4833-96ce-79b2fe2df033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 18 00:51:14.374363 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:14.374330    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:51:14.374763 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:14.374625    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:51:14.375366 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:14.375344    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rwfcw"
Jul 18 00:51:15.173659 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:15.173596    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-zvmsp"
Jul 18 00:51:21.945972 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:21.945927    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"]
Jul 18 00:51:21.952452 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:21.952427    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:21.955915 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:21.955889    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 18 00:51:21.956058 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:21.955889    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 18 00:51:21.956058 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:21.955890    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dl528\""
Jul 18 00:51:21.956058 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:21.955892    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 18 00:51:21.964041 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:21.964012    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"]
Jul 18 00:51:22.117592 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.117544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.117802 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.117608    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gpnn\" (UniqueName: \"kubernetes.io/projected/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-kube-api-access-2gpnn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.117802 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.117665    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.117802 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.117704    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.117802 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.117731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.117936 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.117812    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.219378 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219266    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.219378 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.219378 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219363    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.219749 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219425    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.219749 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219611    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gpnn\" (UniqueName: \"kubernetes.io/projected/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-kube-api-access-2gpnn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.219749 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219690    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.219975 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219951    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.220043 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.219985    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.220181 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.220159    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.221984 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.221956    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.222144 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.222127    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.227527 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.227506    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gpnn\" (UniqueName: \"kubernetes.io/projected/86a3f4a6-ecd6-48aa-b388-f056ab0fe814-kube-api-access-2gpnn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz\" (UID: \"86a3f4a6-ecd6-48aa-b388-f056ab0fe814\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.263487 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.263447    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:22.412214 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:22.412183    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"]
Jul 18 00:51:22.414544 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:51:22.414508    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a3f4a6_ecd6_48aa_b388_f056ab0fe814.slice/crio-ced0eb501fe5892e0c8cbc255247d9cd11e409908f7cfb0a1e171ce9de6e6fcc WatchSource:0}: Error finding container ced0eb501fe5892e0c8cbc255247d9cd11e409908f7cfb0a1e171ce9de6e6fcc: Status 404 returned error can't find the container with id ced0eb501fe5892e0c8cbc255247d9cd11e409908f7cfb0a1e171ce9de6e6fcc
Jul 18 00:51:23.271285 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:23.271242    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz" event={"ID":"86a3f4a6-ecd6-48aa-b388-f056ab0fe814","Type":"ContainerStarted","Data":"ced0eb501fe5892e0c8cbc255247d9cd11e409908f7cfb0a1e171ce9de6e6fcc"}
Jul 18 00:51:24.278597 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:24.278559    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz" event={"ID":"86a3f4a6-ecd6-48aa-b388-f056ab0fe814","Type":"ContainerStarted","Data":"34fcc64e11341586e5c7fc187f03e5a1eef26b14cdc97d7658cb2d554cafa203"}
Jul 18 00:51:24.295788 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:24.295740    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz" podStartSLOduration=2.285370091 podStartE2EDuration="3.295722338s" totalImagesPullingTime="1.010352247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:24.294860261 +0000 UTC m=+640.420795257" watchObservedRunningTime="2026-07-18 00:51:24.295722338 +0000 UTC m=+640.421657317"
Jul 18 00:51:32.264297 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:32.264252    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:32.264861 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:32.264365    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:32.277386 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:32.277354    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:32.328660 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:32.328627    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz"
Jul 18 00:51:42.732753 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.732710    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"]
Jul 18 00:51:42.736744 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.736714    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.739280 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.739256    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 18 00:51:42.746063 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.746033    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"]
Jul 18 00:51:42.819251 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.819213    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.819446 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.819274    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.819446 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.819327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knp9l\" (UniqueName: \"kubernetes.io/projected/6c85eddf-8a21-4304-be86-d1314e06f97d-kube-api-access-knp9l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.819446 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.819351    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85eddf-8a21-4304-be86-d1314e06f97d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.819446 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.819420    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.819446 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.819443    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920332 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920285    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920332 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920335    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920549 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920376    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920549 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920420    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920549 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920468    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knp9l\" (UniqueName: \"kubernetes.io/projected/6c85eddf-8a21-4304-be86-d1314e06f97d-kube-api-access-knp9l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920549 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920506    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85eddf-8a21-4304-be86-d1314e06f97d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920808 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.920912 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920883    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.921033 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.920922    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.922801 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.922779    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c85eddf-8a21-4304-be86-d1314e06f97d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.922983 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.922964    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c85eddf-8a21-4304-be86-d1314e06f97d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:42.928316 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:42.928287    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knp9l\" (UniqueName: \"kubernetes.io/projected/6c85eddf-8a21-4304-be86-d1314e06f97d-kube-api-access-knp9l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j9qwv\" (UID: \"6c85eddf-8a21-4304-be86-d1314e06f97d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:43.048715 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:43.048682    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:43.203059 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:43.202805    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"]
Jul 18 00:51:43.211250 ip-10-0-141-161 kubenswrapper[2554]: W0718 00:51:43.211213    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c85eddf_8a21_4304_be86_d1314e06f97d.slice/crio-6010fd30fe77d211b089466f5ab78a989b15b4a8e64be89f694a8a415450a5a0 WatchSource:0}: Error finding container 6010fd30fe77d211b089466f5ab78a989b15b4a8e64be89f694a8a415450a5a0: Status 404 returned error can't find the container with id 6010fd30fe77d211b089466f5ab78a989b15b4a8e64be89f694a8a415450a5a0
Jul 18 00:51:43.375656 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:43.375559    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv" event={"ID":"6c85eddf-8a21-4304-be86-d1314e06f97d","Type":"ContainerStarted","Data":"6010fd30fe77d211b089466f5ab78a989b15b4a8e64be89f694a8a415450a5a0"}
Jul 18 00:51:44.381452 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:44.381410    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv" event={"ID":"6c85eddf-8a21-4304-be86-d1314e06f97d","Type":"ContainerStarted","Data":"86a46faba0d74057df2a6ed414c2d2fe27141500984e6e77ac5c5f25b666f4b0"}
Jul 18 00:51:44.396938 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:44.396817    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv" podStartSLOduration=2.222625705 podStartE2EDuration="2.39679268s" totalImagesPullingTime="174.166975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:44.396663549 +0000 UTC m=+660.522598529" watchObservedRunningTime="2026-07-18 00:51:44.39679268 +0000 UTC m=+660.522727660"
Jul 18 00:51:53.049291 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:53.049252    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:53.049883 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:53.049312    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:53.062350 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:53.062321    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 00:51:53.438085 ip-10-0-141-161 kubenswrapper[2554]: I0718 00:51:53.438000    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j9qwv"
Jul 18 01:07:41.962424 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:41.962393    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"]
Jul 18 01:07:41.966146 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:41.966127    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:41.970607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:41.970582    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9bd91509-openshift-default-dockercfg-v7586\""
Jul 18 01:07:42.018054 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.018025    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"]
Jul 18 01:07:42.085529 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b887f2f-91f3-48c1-9f01-78d968111008-istiod-ca-cert\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085731 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085564    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pw9\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-kube-api-access-z4pw9\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085731 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085610    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-data\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085731 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085660    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-credential-socket\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085731 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085690    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-socket\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085884 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085783    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-envoy\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085884 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085815    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-certs\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085884 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-istio-token\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.085884 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.085867    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b887f2f-91f3-48c1-9f01-78d968111008-istio-podinfo\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.186988 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.186951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b887f2f-91f3-48c1-9f01-78d968111008-istiod-ca-cert\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187028    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pw9\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-kube-api-access-z4pw9\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187079    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-data\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187105    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-credential-socket\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187286 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187148    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-socket\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187286 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187181    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-envoy\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-certs\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187383    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-istio-token\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187415    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b887f2f-91f3-48c1-9f01-78d968111008-istio-podinfo\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187557 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187504    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-data\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187679 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187656    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-socket\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187760 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-certs\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187852 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187833    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-credential-socket\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.187913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.187832    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b887f2f-91f3-48c1-9f01-78d968111008-istiod-ca-cert\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.189813 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.189789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-envoy\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.190247 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.190200    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b887f2f-91f3-48c1-9f01-78d968111008-istio-podinfo\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.197419 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.197393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pw9\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-kube-api-access-z4pw9\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.198337 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.198308    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-istio-token\") pod \"e2e-ait-9bd91509-openshift-default-67b8859845-txc4d\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") " pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.275658 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.275629    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:42.410891 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.410866    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"]
Jul 18 01:07:42.415475 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:07:42.413195    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b887f2f_91f3_48c1_9f01_78d968111008.slice/crio-54c1298fab8e4ef35b47dfb47e005fd9fe9d6b64a26841e3e458b0f945333a8d WatchSource:0}: Error finding container 54c1298fab8e4ef35b47dfb47e005fd9fe9d6b64a26841e3e458b0f945333a8d: Status 404 returned error can't find the container with id 54c1298fab8e4ef35b47dfb47e005fd9fe9d6b64a26841e3e458b0f945333a8d
Jul 18 01:07:42.417467 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.417431    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:07:42.420656 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.417865    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:07:42.420656 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.418016    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:07:42.960160 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.960123    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" event={"ID":"7b887f2f-91f3-48c1-9f01-78d968111008","Type":"ContainerStarted","Data":"afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af"}
Jul 18 01:07:42.960350 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.960226    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" event={"ID":"7b887f2f-91f3-48c1-9f01-78d968111008","Type":"ContainerStarted","Data":"54c1298fab8e4ef35b47dfb47e005fd9fe9d6b64a26841e3e458b0f945333a8d"}
Jul 18 01:07:42.982953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:42.982589    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" podStartSLOduration=1.982571483 podStartE2EDuration="1.982571483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:07:42.982074279 +0000 UTC m=+1619.108009262" watchObservedRunningTime="2026-07-18 01:07:42.982571483 +0000 UTC m=+1619.108506466"
Jul 18 01:07:43.276550 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:43.276507    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:43.277875 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:43.277821    2554 patch_prober.go:28] interesting pod/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 18 01:07:43.278023 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:43.277892    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 18 01:07:44.279870 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:44.279837    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:44.968938 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:44.968906    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:44.969967 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:44.969947    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:48.191156 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.191113    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"]
Jul 18 01:07:48.194707 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.194687    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"
Jul 18 01:07:48.197996 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.197979    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:07:48.204947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.204888    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"]
Jul 18 01:07:48.352306 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.352272    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22509adc-2d95-44c9-9a4d-ca49ad2907c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r\" (UID: \"22509adc-2d95-44c9-9a4d-ca49ad2907c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"
Jul 18 01:07:48.453258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.453171    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22509adc-2d95-44c9-9a4d-ca49ad2907c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r\" (UID: \"22509adc-2d95-44c9-9a4d-ca49ad2907c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"
Jul 18 01:07:48.453866 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.453846    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22509adc-2d95-44c9-9a4d-ca49ad2907c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r\" (UID: \"22509adc-2d95-44c9-9a4d-ca49ad2907c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"
Jul 18 01:07:48.506060 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.506026    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"
Jul 18 01:07:48.673254 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.673231    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"]
Jul 18 01:07:48.675761 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:07:48.675734    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22509adc_2d95_44c9_9a4d_ca49ad2907c9.slice/crio-3a8ca5a632c772d499e40cb629363d6e18b780089975228f4c7683c848541e70 WatchSource:0}: Error finding container 3a8ca5a632c772d499e40cb629363d6e18b780089975228f4c7683c848541e70: Status 404 returned error can't find the container with id 3a8ca5a632c772d499e40cb629363d6e18b780089975228f4c7683c848541e70
Jul 18 01:07:48.678090 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.678071    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:07:48.991067 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:48.991032    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" event={"ID":"22509adc-2d95-44c9-9a4d-ca49ad2907c9","Type":"ContainerStarted","Data":"3a8ca5a632c772d499e40cb629363d6e18b780089975228f4c7683c848541e70"}
Jul 18 01:07:53.022886 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:53.022843    2554 generic.go:356] "Generic (PLEG): container finished" podID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerID="d7cd3148e3035506c54902368c46b6a83d27280cf504da0c13e799b4cc8139a6" exitCode=7
Jul 18 01:07:53.023351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:53.022936    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" event={"ID":"22509adc-2d95-44c9-9a4d-ca49ad2907c9","Type":"ContainerDied","Data":"d7cd3148e3035506c54902368c46b6a83d27280cf504da0c13e799b4cc8139a6"}
Jul 18 01:07:53.023351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:53.023106    2554 scope.go:117] "RemoveContainer" containerID="d7cd3148e3035506c54902368c46b6a83d27280cf504da0c13e799b4cc8139a6"
Jul 18 01:07:54.030495 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:54.030460    2554 generic.go:356] "Generic (PLEG): container finished" podID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerID="9c0a026685527781c5a35906e99b16dbabda68b74577ac3b6eea241a71d45f36" exitCode=7
Jul 18 01:07:54.030950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:54.030568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" event={"ID":"22509adc-2d95-44c9-9a4d-ca49ad2907c9","Type":"ContainerDied","Data":"9c0a026685527781c5a35906e99b16dbabda68b74577ac3b6eea241a71d45f36"}
Jul 18 01:07:54.030950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:54.030676    2554 scope.go:117] "RemoveContainer" containerID="d7cd3148e3035506c54902368c46b6a83d27280cf504da0c13e799b4cc8139a6"
Jul 18 01:07:54.030950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:54.030748    2554 scope.go:117] "RemoveContainer" containerID="9c0a026685527781c5a35906e99b16dbabda68b74577ac3b6eea241a71d45f36"
Jul 18 01:07:54.031084 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:07:54.031008    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r_odh-ai-gateway-infra(22509adc-2d95-44c9-9a4d-ca49ad2907c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9"
Jul 18 01:07:55.039273 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:55.039241    2554 generic.go:356] "Generic (PLEG): container finished" podID="7b887f2f-91f3-48c1-9f01-78d968111008" containerID="afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af" exitCode=0
Jul 18 01:07:55.039863 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:55.039313    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" event={"ID":"7b887f2f-91f3-48c1-9f01-78d968111008","Type":"ContainerDied","Data":"afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af"}
Jul 18 01:07:55.039936 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:55.039880    2554 scope.go:117] "RemoveContainer" containerID="afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af"
Jul 18 01:07:55.040975 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:55.040947    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:07:55.041053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:55.041028    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:07:55.041089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:55.041062    2554 scope.go:117] "RemoveContainer" containerID="9c0a026685527781c5a35906e99b16dbabda68b74577ac3b6eea241a71d45f36"
Jul 18 01:07:55.041153 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:55.041064    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:07:55.041238 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:07:55.041220    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r_odh-ai-gateway-infra(22509adc-2d95-44c9-9a4d-ca49ad2907c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9"
Jul 18 01:07:56.047339 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:56.047305    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" event={"ID":"7b887f2f-91f3-48c1-9f01-78d968111008","Type":"ContainerStarted","Data":"fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90"}
Jul 18 01:07:56.276278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:56.276238    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:07:56.277484 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:56.277453    2554 patch_prober.go:28] interesting pod/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 18 01:07:56.277654 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:56.277518    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 18 01:07:57.276693 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:57.276659    2554 patch_prober.go:28] interesting pod/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 18 01:07:57.277092 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:57.276716    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 18 01:07:58.276794 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:58.276759    2554 patch_prober.go:28] interesting pod/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 18 01:07:58.277270 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:58.276818    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 18 01:07:59.276975 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:59.276936    2554 patch_prober.go:28] interesting pod/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 18 01:07:59.277363 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:59.277025    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 18 01:07:59.968819 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:07:59.968792    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:08:00.280171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:00.280144    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:08:00.281189 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:00.281167    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:08:09.331206 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:09.331170    2554 scope.go:117] "RemoveContainer" containerID="9c0a026685527781c5a35906e99b16dbabda68b74577ac3b6eea241a71d45f36"
Jul 18 01:08:10.122721 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:10.122688    2554 generic.go:356] "Generic (PLEG): container finished" podID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerID="5dca072069ecdc52821a57eccd7a502b00b6fb505835007240d35557a87d6a64" exitCode=7
Jul 18 01:08:10.122890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:10.122764    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" event={"ID":"22509adc-2d95-44c9-9a4d-ca49ad2907c9","Type":"ContainerDied","Data":"5dca072069ecdc52821a57eccd7a502b00b6fb505835007240d35557a87d6a64"}
Jul 18 01:08:10.122890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:10.122824    2554 scope.go:117] "RemoveContainer" containerID="9c0a026685527781c5a35906e99b16dbabda68b74577ac3b6eea241a71d45f36"
Jul 18 01:08:10.123208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:10.123181    2554 scope.go:117] "RemoveContainer" containerID="5dca072069ecdc52821a57eccd7a502b00b6fb505835007240d35557a87d6a64"
Jul 18 01:08:10.123417 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:08:10.123395    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r_odh-ai-gateway-infra(22509adc-2d95-44c9-9a4d-ca49ad2907c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9"
Jul 18 01:08:11.172572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:11.172539    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"]
Jul 18 01:08:11.311421 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:11.311393    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"
Jul 18 01:08:11.381909 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:11.381879    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22509adc-2d95-44c9-9a4d-ca49ad2907c9-maas-api-service-ca\") pod \"22509adc-2d95-44c9-9a4d-ca49ad2907c9\" (UID: \"22509adc-2d95-44c9-9a4d-ca49ad2907c9\") "
Jul 18 01:08:11.382206 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:11.382186    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22509adc-2d95-44c9-9a4d-ca49ad2907c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22509adc-2d95-44c9-9a4d-ca49ad2907c9" (UID: "22509adc-2d95-44c9-9a4d-ca49ad2907c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:08:11.482607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:11.482531    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22509adc-2d95-44c9-9a4d-ca49ad2907c9-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:08:12.134600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:12.134565    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r" event={"ID":"22509adc-2d95-44c9-9a4d-ca49ad2907c9","Type":"ContainerDied","Data":"3a8ca5a632c772d499e40cb629363d6e18b780089975228f4c7683c848541e70"}
Jul 18 01:08:12.134836 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:12.134602    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"
Jul 18 01:08:12.134836 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:12.134640    2554 scope.go:117] "RemoveContainer" containerID="5dca072069ecdc52821a57eccd7a502b00b6fb505835007240d35557a87d6a64"
Jul 18 01:08:12.154306 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:12.154279    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"]
Jul 18 01:08:12.157990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:12.157964    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-t2t4r"]
Jul 18 01:08:12.336206 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:12.336177    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" path="/var/lib/kubelet/pods/22509adc-2d95-44c9-9a4d-ca49ad2907c9/volumes"
Jul 18 01:08:48.231595 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.231559    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"]
Jul 18 01:08:48.232026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.231961    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:08:48.232026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.231973    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:08:48.232026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.231984    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:08:48.232026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.231990    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:08:48.232160 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.232071    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:08:48.232160 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.232081    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:08:48.235288 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.235270    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"
Jul 18 01:08:48.237725 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.237702    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:08:48.242490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.242460    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"]
Jul 18 01:08:48.246992 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.246969    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf5b53-3091-4159-b6b5-43737e6c879b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-db2db\" (UID: \"43cf5b53-3091-4159-b6b5-43737e6c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"
Jul 18 01:08:48.347438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.347406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf5b53-3091-4159-b6b5-43737e6c879b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-db2db\" (UID: \"43cf5b53-3091-4159-b6b5-43737e6c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"
Jul 18 01:08:48.348035 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.348017    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf5b53-3091-4159-b6b5-43737e6c879b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-db2db\" (UID: \"43cf5b53-3091-4159-b6b5-43737e6c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"
Jul 18 01:08:48.547580 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.547542    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"
Jul 18 01:08:48.677071 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:48.677049    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"]
Jul 18 01:08:48.679665 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:08:48.679604    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cf5b53_3091_4159_b6b5_43737e6c879b.slice/crio-1008f762dc236f3998e1da71aa5579b852c4af92d86a80dc218208ebf456fa2f WatchSource:0}: Error finding container 1008f762dc236f3998e1da71aa5579b852c4af92d86a80dc218208ebf456fa2f: Status 404 returned error can't find the container with id 1008f762dc236f3998e1da71aa5579b852c4af92d86a80dc218208ebf456fa2f
Jul 18 01:08:49.313729 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:49.313698    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" event={"ID":"43cf5b53-3091-4159-b6b5-43737e6c879b","Type":"ContainerStarted","Data":"b6b104644bea1807ef4c6407b3611ad854ecbf2af3c511f5e83c4e7a0d3a622e"}
Jul 18 01:08:49.313998 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:49.313753    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" event={"ID":"43cf5b53-3091-4159-b6b5-43737e6c879b","Type":"ContainerStarted","Data":"1008f762dc236f3998e1da71aa5579b852c4af92d86a80dc218208ebf456fa2f"}
Jul 18 01:08:49.328846 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:49.328803    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" podStartSLOduration=1.328787235 podStartE2EDuration="1.328787235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:08:49.328686029 +0000 UTC m=+1685.454621009" watchObservedRunningTime="2026-07-18 01:08:49.328787235 +0000 UTC m=+1685.454722213"
Jul 18 01:08:50.320061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:50.320021    2554 generic.go:356] "Generic (PLEG): container finished" podID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerID="b6b104644bea1807ef4c6407b3611ad854ecbf2af3c511f5e83c4e7a0d3a622e" exitCode=7
Jul 18 01:08:50.320445 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:50.320105    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" event={"ID":"43cf5b53-3091-4159-b6b5-43737e6c879b","Type":"ContainerDied","Data":"b6b104644bea1807ef4c6407b3611ad854ecbf2af3c511f5e83c4e7a0d3a622e"}
Jul 18 01:08:50.320445 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:50.320269    2554 scope.go:117] "RemoveContainer" containerID="b6b104644bea1807ef4c6407b3611ad854ecbf2af3c511f5e83c4e7a0d3a622e"
Jul 18 01:08:51.326764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:51.326677    2554 generic.go:356] "Generic (PLEG): container finished" podID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerID="ec6646ff3d2fdce425c4144b240dbd3fd25e0bd45e2a0860d2e53627d1693d32" exitCode=7
Jul 18 01:08:51.327155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:51.326758    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" event={"ID":"43cf5b53-3091-4159-b6b5-43737e6c879b","Type":"ContainerDied","Data":"ec6646ff3d2fdce425c4144b240dbd3fd25e0bd45e2a0860d2e53627d1693d32"}
Jul 18 01:08:51.327155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:51.326814    2554 scope.go:117] "RemoveContainer" containerID="b6b104644bea1807ef4c6407b3611ad854ecbf2af3c511f5e83c4e7a0d3a622e"
Jul 18 01:08:51.327155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:51.326970    2554 scope.go:117] "RemoveContainer" containerID="ec6646ff3d2fdce425c4144b240dbd3fd25e0bd45e2a0860d2e53627d1693d32"
Jul 18 01:08:51.327294 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:08:51.327198    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-db2db_odh-ai-gateway-infra(43cf5b53-3091-4159-b6b5-43737e6c879b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b"
Jul 18 01:08:52.334067 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:08:52.334039    2554 scope.go:117] "RemoveContainer" containerID="ec6646ff3d2fdce425c4144b240dbd3fd25e0bd45e2a0860d2e53627d1693d32"
Jul 18 01:08:52.334467 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:08:52.334252    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-db2db_odh-ai-gateway-infra(43cf5b53-3091-4159-b6b5-43737e6c879b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b"
Jul 18 01:09:07.330455 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:07.330424    2554 scope.go:117] "RemoveContainer" containerID="ec6646ff3d2fdce425c4144b240dbd3fd25e0bd45e2a0860d2e53627d1693d32"
Jul 18 01:09:08.410494 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:08.410409    2554 generic.go:356] "Generic (PLEG): container finished" podID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerID="187df0fe848f9062a02229b8af2dfdddb1b94a36302bca966bedbd1a76589e25" exitCode=7
Jul 18 01:09:08.410921 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:08.410488    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" event={"ID":"43cf5b53-3091-4159-b6b5-43737e6c879b","Type":"ContainerDied","Data":"187df0fe848f9062a02229b8af2dfdddb1b94a36302bca966bedbd1a76589e25"}
Jul 18 01:09:08.410921 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:08.410558    2554 scope.go:117] "RemoveContainer" containerID="ec6646ff3d2fdce425c4144b240dbd3fd25e0bd45e2a0860d2e53627d1693d32"
Jul 18 01:09:08.410921 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:08.410901    2554 scope.go:117] "RemoveContainer" containerID="187df0fe848f9062a02229b8af2dfdddb1b94a36302bca966bedbd1a76589e25"
Jul 18 01:09:08.411153 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:09:08.411133    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-db2db_odh-ai-gateway-infra(43cf5b53-3091-4159-b6b5-43737e6c879b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b"
Jul 18 01:09:09.435882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:09.435846    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"]
Jul 18 01:09:09.573221 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:09.573196    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"
Jul 18 01:09:09.650671 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:09.650638    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf5b53-3091-4159-b6b5-43737e6c879b-maas-api-service-ca\") pod \"43cf5b53-3091-4159-b6b5-43737e6c879b\" (UID: \"43cf5b53-3091-4159-b6b5-43737e6c879b\") "
Jul 18 01:09:09.650990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:09.650967    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cf5b53-3091-4159-b6b5-43737e6c879b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43cf5b53-3091-4159-b6b5-43737e6c879b" (UID: "43cf5b53-3091-4159-b6b5-43737e6c879b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:09:09.752085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:09.752055    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cf5b53-3091-4159-b6b5-43737e6c879b-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:09:10.421859 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:10.421823    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db" event={"ID":"43cf5b53-3091-4159-b6b5-43737e6c879b","Type":"ContainerDied","Data":"1008f762dc236f3998e1da71aa5579b852c4af92d86a80dc218208ebf456fa2f"}
Jul 18 01:09:10.421859 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:10.421854    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"
Jul 18 01:09:10.422042 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:10.421888    2554 scope.go:117] "RemoveContainer" containerID="187df0fe848f9062a02229b8af2dfdddb1b94a36302bca966bedbd1a76589e25"
Jul 18 01:09:10.438588 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:10.438562    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"]
Jul 18 01:09:10.442080 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:10.442054    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-db2db"]
Jul 18 01:09:12.336662 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:12.336629    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" path="/var/lib/kubelet/pods/43cf5b53-3091-4159-b6b5-43737e6c879b/volumes"
Jul 18 01:09:48.277676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.277643    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"]
Jul 18 01:09:48.278089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278025    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:09:48.278089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278036    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:09:48.278089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278046    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:09:48.278089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278052    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:09:48.278089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278061    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:09:48.278089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278066    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:09:48.278279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278136    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:09:48.278279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278147    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="22509adc-2d95-44c9-9a4d-ca49ad2907c9" containerName="revoke-keys"
Jul 18 01:09:48.278279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.278156    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:09:48.280972 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.280955    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"
Jul 18 01:09:48.283336 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.283318    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:09:48.287257 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.287232    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"]
Jul 18 01:09:48.425984 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.425948    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c61ee-70bb-4036-aca8-e1099b0d9469-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-m4djk\" (UID: \"f29c61ee-70bb-4036-aca8-e1099b0d9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"
Jul 18 01:09:48.531772 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.527937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c61ee-70bb-4036-aca8-e1099b0d9469-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-m4djk\" (UID: \"f29c61ee-70bb-4036-aca8-e1099b0d9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"
Jul 18 01:09:48.531772 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.528851    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c61ee-70bb-4036-aca8-e1099b0d9469-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-m4djk\" (UID: \"f29c61ee-70bb-4036-aca8-e1099b0d9469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"
Jul 18 01:09:48.595446 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.595416    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"
Jul 18 01:09:48.732077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:48.732049    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"]
Jul 18 01:09:48.734651 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:09:48.734607    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29c61ee_70bb_4036_aca8_e1099b0d9469.slice/crio-9a51a0e83ab1bd2d158e3733dfd9c86117775c4e7bad6800048a9ed95849a13b WatchSource:0}: Error finding container 9a51a0e83ab1bd2d158e3733dfd9c86117775c4e7bad6800048a9ed95849a13b: Status 404 returned error can't find the container with id 9a51a0e83ab1bd2d158e3733dfd9c86117775c4e7bad6800048a9ed95849a13b
Jul 18 01:09:49.620012 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:49.619910    2554 generic.go:356] "Generic (PLEG): container finished" podID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerID="fcdf633f79bd706e863420d1c2c7e03734dc5df2f233337a7924692bc26aba00" exitCode=7
Jul 18 01:09:49.620379 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:49.620003    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" event={"ID":"f29c61ee-70bb-4036-aca8-e1099b0d9469","Type":"ContainerDied","Data":"fcdf633f79bd706e863420d1c2c7e03734dc5df2f233337a7924692bc26aba00"}
Jul 18 01:09:49.620379 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:49.620056    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" event={"ID":"f29c61ee-70bb-4036-aca8-e1099b0d9469","Type":"ContainerStarted","Data":"9a51a0e83ab1bd2d158e3733dfd9c86117775c4e7bad6800048a9ed95849a13b"}
Jul 18 01:09:49.620379 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:49.620182    2554 scope.go:117] "RemoveContainer" containerID="fcdf633f79bd706e863420d1c2c7e03734dc5df2f233337a7924692bc26aba00"
Jul 18 01:09:50.627829 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:50.627734    2554 generic.go:356] "Generic (PLEG): container finished" podID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerID="f55514c6608e530a3e7d51a090b1689a0545b9a2723aabab917625ab2e4476c5" exitCode=7
Jul 18 01:09:50.627829 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:50.627800    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" event={"ID":"f29c61ee-70bb-4036-aca8-e1099b0d9469","Type":"ContainerDied","Data":"f55514c6608e530a3e7d51a090b1689a0545b9a2723aabab917625ab2e4476c5"}
Jul 18 01:09:50.628305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:50.627860    2554 scope.go:117] "RemoveContainer" containerID="fcdf633f79bd706e863420d1c2c7e03734dc5df2f233337a7924692bc26aba00"
Jul 18 01:09:50.628305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:50.628083    2554 scope.go:117] "RemoveContainer" containerID="f55514c6608e530a3e7d51a090b1689a0545b9a2723aabab917625ab2e4476c5"
Jul 18 01:09:50.628383 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:09:50.628327    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-m4djk_odh-ai-gateway-infra(f29c61ee-70bb-4036-aca8-e1099b0d9469)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469"
Jul 18 01:09:51.634994 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:09:51.634966    2554 scope.go:117] "RemoveContainer" containerID="f55514c6608e530a3e7d51a090b1689a0545b9a2723aabab917625ab2e4476c5"
Jul 18 01:09:51.635365 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:09:51.635202    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-m4djk_odh-ai-gateway-infra(f29c61ee-70bb-4036-aca8-e1099b0d9469)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469"
Jul 18 01:10:04.334208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:04.334174    2554 scope.go:117] "RemoveContainer" containerID="f55514c6608e530a3e7d51a090b1689a0545b9a2723aabab917625ab2e4476c5"
Jul 18 01:10:04.701427 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:04.701388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" event={"ID":"f29c61ee-70bb-4036-aca8-e1099b0d9469","Type":"ContainerStarted","Data":"5fed77f1cb0c46fef7aa9b19d9c5f6e59656cb4789ca81008b2a256836f42859"}
Jul 18 01:10:04.721872 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:04.721803    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" podStartSLOduration=16.72178178 podStartE2EDuration="16.72178178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:10:04.719880457 +0000 UTC m=+1760.845815437" watchObservedRunningTime="2026-07-18 01:10:04.72178178 +0000 UTC m=+1760.847716760"
Jul 18 01:10:05.708145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:05.708113    2554 generic.go:356] "Generic (PLEG): container finished" podID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerID="5fed77f1cb0c46fef7aa9b19d9c5f6e59656cb4789ca81008b2a256836f42859" exitCode=7
Jul 18 01:10:05.708575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:05.708188    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" event={"ID":"f29c61ee-70bb-4036-aca8-e1099b0d9469","Type":"ContainerDied","Data":"5fed77f1cb0c46fef7aa9b19d9c5f6e59656cb4789ca81008b2a256836f42859"}
Jul 18 01:10:05.708575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:05.708262    2554 scope.go:117] "RemoveContainer" containerID="f55514c6608e530a3e7d51a090b1689a0545b9a2723aabab917625ab2e4476c5"
Jul 18 01:10:05.708694 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:05.708577    2554 scope.go:117] "RemoveContainer" containerID="5fed77f1cb0c46fef7aa9b19d9c5f6e59656cb4789ca81008b2a256836f42859"
Jul 18 01:10:05.708847 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:10:05.708823    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-m4djk_odh-ai-gateway-infra(f29c61ee-70bb-4036-aca8-e1099b0d9469)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469"
Jul 18 01:10:05.730400 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:05.730367    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"]
Jul 18 01:10:06.848190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:06.848163    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"
Jul 18 01:10:06.903200 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:06.903166    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c61ee-70bb-4036-aca8-e1099b0d9469-maas-api-service-ca\") pod \"f29c61ee-70bb-4036-aca8-e1099b0d9469\" (UID: \"f29c61ee-70bb-4036-aca8-e1099b0d9469\") "
Jul 18 01:10:06.903553 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:06.903525    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29c61ee-70bb-4036-aca8-e1099b0d9469-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f29c61ee-70bb-4036-aca8-e1099b0d9469" (UID: "f29c61ee-70bb-4036-aca8-e1099b0d9469"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:10:07.004474 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:07.004434    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29c61ee-70bb-4036-aca8-e1099b0d9469-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:07.720822 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:07.720788    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk" event={"ID":"f29c61ee-70bb-4036-aca8-e1099b0d9469","Type":"ContainerDied","Data":"9a51a0e83ab1bd2d158e3733dfd9c86117775c4e7bad6800048a9ed95849a13b"}
Jul 18 01:10:07.720822 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:07.720816    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"
Jul 18 01:10:07.721088 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:07.720856    2554 scope.go:117] "RemoveContainer" containerID="5fed77f1cb0c46fef7aa9b19d9c5f6e59656cb4789ca81008b2a256836f42859"
Jul 18 01:10:07.742165 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:07.742135    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"]
Jul 18 01:10:07.745780 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:07.745756    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-m4djk"]
Jul 18 01:10:08.338388 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:08.338350    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" path="/var/lib/kubelet/pods/f29c61ee-70bb-4036-aca8-e1099b0d9469/volumes"
Jul 18 01:10:38.317861 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.317770    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"]
Jul 18 01:10:38.318463 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318393    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:10:38.318463 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318413    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:10:38.318463 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318429    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:10:38.318463 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318438    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:10:38.318737 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318469    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:10:38.318737 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318478    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:10:38.318737 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318570    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:10:38.318737 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318584    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:10:38.318737 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.318595    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cf5b53-3091-4159-b6b5-43737e6c879b" containerName="revoke-keys"
Jul 18 01:10:38.321719 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.321698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"
Jul 18 01:10:38.324024 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.324004    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:10:38.327916 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.327893    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"]
Jul 18 01:10:38.412448 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.412414    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bce5465-c803-4bb0-8048-c89f689bb641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp\" (UID: \"7bce5465-c803-4bb0-8048-c89f689bb641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"
Jul 18 01:10:38.513846 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.513812    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bce5465-c803-4bb0-8048-c89f689bb641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp\" (UID: \"7bce5465-c803-4bb0-8048-c89f689bb641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"
Jul 18 01:10:38.514474 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.514450    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bce5465-c803-4bb0-8048-c89f689bb641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp\" (UID: \"7bce5465-c803-4bb0-8048-c89f689bb641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"
Jul 18 01:10:38.634396 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.634312    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"
Jul 18 01:10:38.760033 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.760005    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"]
Jul 18 01:10:38.762708 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:10:38.762673    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bce5465_c803_4bb0_8048_c89f689bb641.slice/crio-194fc354eeee719c18127082389837a2ce79f8d43fa4e0172167722f34bd9256 WatchSource:0}: Error finding container 194fc354eeee719c18127082389837a2ce79f8d43fa4e0172167722f34bd9256: Status 404 returned error can't find the container with id 194fc354eeee719c18127082389837a2ce79f8d43fa4e0172167722f34bd9256
Jul 18 01:10:38.875302 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:38.875267    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" event={"ID":"7bce5465-c803-4bb0-8048-c89f689bb641","Type":"ContainerStarted","Data":"194fc354eeee719c18127082389837a2ce79f8d43fa4e0172167722f34bd9256"}
Jul 18 01:10:39.882175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:39.882142    2554 generic.go:356] "Generic (PLEG): container finished" podID="7bce5465-c803-4bb0-8048-c89f689bb641" containerID="30fa612d04fad414158eb867648f57e19409ed85da229274a2a2ef9d011a2079" exitCode=7
Jul 18 01:10:39.882586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:39.882227    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" event={"ID":"7bce5465-c803-4bb0-8048-c89f689bb641","Type":"ContainerDied","Data":"30fa612d04fad414158eb867648f57e19409ed85da229274a2a2ef9d011a2079"}
Jul 18 01:10:39.882586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:39.882376    2554 scope.go:117] "RemoveContainer" containerID="30fa612d04fad414158eb867648f57e19409ed85da229274a2a2ef9d011a2079"
Jul 18 01:10:40.888534 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:40.888442    2554 generic.go:356] "Generic (PLEG): container finished" podID="7bce5465-c803-4bb0-8048-c89f689bb641" containerID="7cae6828223a36fa777ed5030699084dbc69eecf129370f08dcef1e4bb23ea09" exitCode=7
Jul 18 01:10:40.888534 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:40.888493    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" event={"ID":"7bce5465-c803-4bb0-8048-c89f689bb641","Type":"ContainerDied","Data":"7cae6828223a36fa777ed5030699084dbc69eecf129370f08dcef1e4bb23ea09"}
Jul 18 01:10:40.889002 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:40.888551    2554 scope.go:117] "RemoveContainer" containerID="30fa612d04fad414158eb867648f57e19409ed85da229274a2a2ef9d011a2079"
Jul 18 01:10:40.889002 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:40.888743    2554 scope.go:117] "RemoveContainer" containerID="7cae6828223a36fa777ed5030699084dbc69eecf129370f08dcef1e4bb23ea09"
Jul 18 01:10:40.889002 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:10:40.888974    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp_odh-ai-gateway-infra(7bce5465-c803-4bb0-8048-c89f689bb641)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" podUID="7bce5465-c803-4bb0-8048-c89f689bb641"
Jul 18 01:10:41.900421 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:41.900388    2554 scope.go:117] "RemoveContainer" containerID="7cae6828223a36fa777ed5030699084dbc69eecf129370f08dcef1e4bb23ea09"
Jul 18 01:10:41.900843 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:10:41.900646    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp_odh-ai-gateway-infra(7bce5465-c803-4bb0-8048-c89f689bb641)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" podUID="7bce5465-c803-4bb0-8048-c89f689bb641"
Jul 18 01:10:48.300753 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:48.300709    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"]
Jul 18 01:10:48.301281 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:48.300979    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy" containerID="cri-o://fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90" gracePeriod=30
Jul 18 01:10:53.563838 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.563776    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:10:53.658640 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658591    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b887f2f-91f3-48c1-9f01-78d968111008-istiod-ca-cert\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.658804 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658646    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4pw9\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-kube-api-access-z4pw9\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.658804 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658668    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-data\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.658804 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658696    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-socket\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.658804 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658724    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-envoy\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.658804 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658783    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b887f2f-91f3-48c1-9f01-78d968111008-istio-podinfo\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.659076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658834    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-istio-token\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.659076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658880    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-credential-socket\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.659076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.658947    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-certs\") pod \"7b887f2f-91f3-48c1-9f01-78d968111008\" (UID: \"7b887f2f-91f3-48c1-9f01-78d968111008\") "
Jul 18 01:10:53.659236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659088    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b887f2f-91f3-48c1-9f01-78d968111008-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:10:53.659357 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659338    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:10:53.659424 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659353    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:10:53.659424 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659377    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:10:53.659564 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659546    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.659602 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659571    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.659602 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659586    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b887f2f-91f3-48c1-9f01-78d968111008-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.659686 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659599    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.659686 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.659554    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:10:53.661115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.661086    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b887f2f-91f3-48c1-9f01-78d968111008-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:10:53.661323 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.661280    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:10:53.661323 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.661301    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-kube-api-access-z4pw9" (OuterVolumeSpecName: "kube-api-access-z4pw9") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "kube-api-access-z4pw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:10:53.661323 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.661291    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b887f2f-91f3-48c1-9f01-78d968111008" (UID: "7b887f2f-91f3-48c1-9f01-78d968111008"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:10:53.760862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.760826    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4pw9\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-kube-api-access-z4pw9\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.760862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.760855    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.760862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.760865    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b887f2f-91f3-48c1-9f01-78d968111008-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.760862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.760873    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b887f2f-91f3-48c1-9f01-78d968111008-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.761126 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.760880    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b887f2f-91f3-48c1-9f01-78d968111008-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:53.961016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.960935    2554 generic.go:356] "Generic (PLEG): container finished" podID="7b887f2f-91f3-48c1-9f01-78d968111008" containerID="fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90" exitCode=0
Jul 18 01:10:53.961016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.960990    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"
Jul 18 01:10:53.961221 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.961025    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" event={"ID":"7b887f2f-91f3-48c1-9f01-78d968111008","Type":"ContainerDied","Data":"fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90"}
Jul 18 01:10:53.961221 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.961080    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d" event={"ID":"7b887f2f-91f3-48c1-9f01-78d968111008","Type":"ContainerDied","Data":"54c1298fab8e4ef35b47dfb47e005fd9fe9d6b64a26841e3e458b0f945333a8d"}
Jul 18 01:10:53.961221 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.961101    2554 scope.go:117] "RemoveContainer" containerID="fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90"
Jul 18 01:10:53.970220 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.970201    2554 scope.go:117] "RemoveContainer" containerID="afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af"
Jul 18 01:10:53.978017 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.977997    2554 scope.go:117] "RemoveContainer" containerID="fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90"
Jul 18 01:10:53.978267 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:10:53.978245    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90\": container with ID starting with fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90 not found: ID does not exist" containerID="fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90"
Jul 18 01:10:53.978330 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.978275    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90"} err="failed to get container status \"fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90\": rpc error: code = NotFound desc = could not find container \"fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90\": container with ID starting with fd09a1568450ff164d2de885ed871a74d4810ebb8dea6e368131d697d4a7be90 not found: ID does not exist"
Jul 18 01:10:53.978330 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.978293    2554 scope.go:117] "RemoveContainer" containerID="afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af"
Jul 18 01:10:53.978499 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:10:53.978485    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af\": container with ID starting with afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af not found: ID does not exist" containerID="afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af"
Jul 18 01:10:53.978543 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.978501    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af"} err="failed to get container status \"afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af\": rpc error: code = NotFound desc = could not find container \"afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af\": container with ID starting with afd2134e94c05e3c73b9c5a7194d93a3b357d77016ba753cea8784f80a5591af not found: ID does not exist"
Jul 18 01:10:53.990902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:53.990880    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"]
Jul 18 01:10:54.000787 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:54.000765    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9bd91509-openshift-default-67b8859845-txc4d"]
Jul 18 01:10:54.336817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:54.336787    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" path="/var/lib/kubelet/pods/7b887f2f-91f3-48c1-9f01-78d968111008/volumes"
Jul 18 01:10:56.331012 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:56.330985    2554 scope.go:117] "RemoveContainer" containerID="7cae6828223a36fa777ed5030699084dbc69eecf129370f08dcef1e4bb23ea09"
Jul 18 01:10:56.977462 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:56.977423    2554 generic.go:356] "Generic (PLEG): container finished" podID="7bce5465-c803-4bb0-8048-c89f689bb641" containerID="870e27b2b41fa3a21aac7989cc49b2e72349fe8c48c8d5bd6d51c3c9664cadbd" exitCode=6
Jul 18 01:10:56.977645 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:56.977472    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" event={"ID":"7bce5465-c803-4bb0-8048-c89f689bb641","Type":"ContainerDied","Data":"870e27b2b41fa3a21aac7989cc49b2e72349fe8c48c8d5bd6d51c3c9664cadbd"}
Jul 18 01:10:56.977645 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:56.977549    2554 scope.go:117] "RemoveContainer" containerID="7cae6828223a36fa777ed5030699084dbc69eecf129370f08dcef1e4bb23ea09"
Jul 18 01:10:56.977892 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:56.977869    2554 scope.go:117] "RemoveContainer" containerID="870e27b2b41fa3a21aac7989cc49b2e72349fe8c48c8d5bd6d51c3c9664cadbd"
Jul 18 01:10:56.978132 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:10:56.978113    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp_odh-ai-gateway-infra(7bce5465-c803-4bb0-8048-c89f689bb641)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" podUID="7bce5465-c803-4bb0-8048-c89f689bb641"
Jul 18 01:10:58.002533 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.002496    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"]
Jul 18 01:10:58.144485 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.144460    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"
Jul 18 01:10:58.201498 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.201460    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bce5465-c803-4bb0-8048-c89f689bb641-maas-api-service-ca\") pod \"7bce5465-c803-4bb0-8048-c89f689bb641\" (UID: \"7bce5465-c803-4bb0-8048-c89f689bb641\") "
Jul 18 01:10:58.201861 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.201837    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bce5465-c803-4bb0-8048-c89f689bb641-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bce5465-c803-4bb0-8048-c89f689bb641" (UID: "7bce5465-c803-4bb0-8048-c89f689bb641"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:10:58.303022 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.302934    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bce5465-c803-4bb0-8048-c89f689bb641-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:10:58.990288 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.990263    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"
Jul 18 01:10:58.990462 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.990300    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp" event={"ID":"7bce5465-c803-4bb0-8048-c89f689bb641","Type":"ContainerDied","Data":"194fc354eeee719c18127082389837a2ce79f8d43fa4e0172167722f34bd9256"}
Jul 18 01:10:58.990462 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:58.990362    2554 scope.go:117] "RemoveContainer" containerID="870e27b2b41fa3a21aac7989cc49b2e72349fe8c48c8d5bd6d51c3c9664cadbd"
Jul 18 01:10:59.005936 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:59.005913    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"]
Jul 18 01:10:59.007581 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:10:59.007563    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8lgjp"]
Jul 18 01:11:00.336460 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.336425    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" path="/var/lib/kubelet/pods/7bce5465-c803-4bb0-8048-c89f689bb641/volumes"
Jul 18 01:11:00.609736 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.609648    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"]
Jul 18 01:11:00.610253 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610236    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.610302 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610257    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.610302 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610276    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:11:00.610302 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610285    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:11:00.610302 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610298    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.610430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610307    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.610430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610320    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy"
Jul 18 01:11:00.610430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610328    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy"
Jul 18 01:11:00.610430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610345    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.610430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610353    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.610430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610366    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy"
Jul 18 01:11:00.610430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610374    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy"
Jul 18 01:11:00.610661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610455    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29c61ee-70bb-4036-aca8-e1099b0d9469" containerName="revoke-keys"
Jul 18 01:11:00.610661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610466    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy"
Jul 18 01:11:00.610661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610477    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.610661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610487    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b887f2f-91f3-48c1-9f01-78d968111008" containerName="istio-proxy"
Jul 18 01:11:00.610661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.610497    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:00.615295 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.615273    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.618611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.618586    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bf4f6c8d-gw-openshift-default-dockercfg-rchzj\""
Jul 18 01:11:00.626254 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.626230    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"]
Jul 18 01:11:00.725356 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725317    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-socket\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725356 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725367    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-istio-token\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725404    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-data\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725444    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e9a9e2-9034-4481-a00c-31990de5d687-istio-podinfo\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725530    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-certs\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725696 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-credential-socket\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725696 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725685    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxxg\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-kube-api-access-zlxxg\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725761 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725739    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e9a9e2-9034-4481-a00c-31990de5d687-istiod-ca-cert\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.725794 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.725757    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-envoy\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.826735 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.826694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e9a9e2-9034-4481-a00c-31990de5d687-istio-podinfo\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.826907 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.826749    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-certs\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.826955 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.826924    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-credential-socket\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827067 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxxg\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-kube-api-access-zlxxg\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827088    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-certs\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827097    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e9a9e2-9034-4481-a00c-31990de5d687-istiod-ca-cert\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827125    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-envoy\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827170    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-socket\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827193    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-istio-token\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827511 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827221    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-data\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827511 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827251    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-credential-socket\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827649 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827511    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-socket\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827649 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827555    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-data\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.827782 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.827712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e9a9e2-9034-4481-a00c-31990de5d687-istiod-ca-cert\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.829290 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.829266    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-envoy\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.829526 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.829509    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e9a9e2-9034-4481-a00c-31990de5d687-istio-podinfo\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.836797 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.836775    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-istio-token\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.837100 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.837080    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxxg\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-kube-api-access-zlxxg\") pod \"e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") " pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:00.930111 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:00.930030    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:01.077322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:01.077261    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"]
Jul 18 01:11:01.080094 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:11:01.080060    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e9a9e2_9034_4481_a00c_31990de5d687.slice/crio-934b1111f0d4705d3cb051164bffb9967dda9a6d47a6b5b49aae7facdfa5c883 WatchSource:0}: Error finding container 934b1111f0d4705d3cb051164bffb9967dda9a6d47a6b5b49aae7facdfa5c883: Status 404 returned error can't find the container with id 934b1111f0d4705d3cb051164bffb9967dda9a6d47a6b5b49aae7facdfa5c883
Jul 18 01:11:01.082343 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:01.082313    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:11:01.082454 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:01.082393    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:11:01.082454 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:01.082435    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:11:02.010979 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:02.010929    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" event={"ID":"27e9a9e2-9034-4481-a00c-31990de5d687","Type":"ContainerStarted","Data":"17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87"}
Jul 18 01:11:02.011386 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:02.011013    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" event={"ID":"27e9a9e2-9034-4481-a00c-31990de5d687","Type":"ContainerStarted","Data":"934b1111f0d4705d3cb051164bffb9967dda9a6d47a6b5b49aae7facdfa5c883"}
Jul 18 01:11:02.034834 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:02.034782    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" podStartSLOduration=2.034764654 podStartE2EDuration="2.034764654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:11:02.03238595 +0000 UTC m=+1818.158320925" watchObservedRunningTime="2026-07-18 01:11:02.034764654 +0000 UTC m=+1818.160699633"
Jul 18 01:11:02.930853 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:02.930817    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:02.935408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:02.935382    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:03.016378 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:03.016342    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:03.017255 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:03.017234    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:06.371384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.371349    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"]
Jul 18 01:11:06.371880 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.371865    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bce5465-c803-4bb0-8048-c89f689bb641" containerName="revoke-keys"
Jul 18 01:11:06.375129 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.375112    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"
Jul 18 01:11:06.377916 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.377898    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:11:06.381636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.381594    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"]
Jul 18 01:11:06.481832 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.481798    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f27b62-645f-4387-ab8a-4833d3baa022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7\" (UID: \"e1f27b62-645f-4387-ab8a-4833d3baa022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"
Jul 18 01:11:06.582774 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.582720    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f27b62-645f-4387-ab8a-4833d3baa022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7\" (UID: \"e1f27b62-645f-4387-ab8a-4833d3baa022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"
Jul 18 01:11:06.583370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.583348    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f27b62-645f-4387-ab8a-4833d3baa022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7\" (UID: \"e1f27b62-645f-4387-ab8a-4833d3baa022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"
Jul 18 01:11:06.687511 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.687411    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"
Jul 18 01:11:06.872195 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:06.872167    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"]
Jul 18 01:11:06.874481 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:11:06.874449    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f27b62_645f_4387_ab8a_4833d3baa022.slice/crio-5dcf61791f3111f0b8d9c9ada0f83fe90c4d7672d7504c05d79301f5f8459ff8 WatchSource:0}: Error finding container 5dcf61791f3111f0b8d9c9ada0f83fe90c4d7672d7504c05d79301f5f8459ff8: Status 404 returned error can't find the container with id 5dcf61791f3111f0b8d9c9ada0f83fe90c4d7672d7504c05d79301f5f8459ff8
Jul 18 01:11:07.039297 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:07.039261    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" event={"ID":"e1f27b62-645f-4387-ab8a-4833d3baa022","Type":"ContainerStarted","Data":"5dcf61791f3111f0b8d9c9ada0f83fe90c4d7672d7504c05d79301f5f8459ff8"}
Jul 18 01:11:08.046147 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:08.046112    2554 generic.go:356] "Generic (PLEG): container finished" podID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerID="a46669a2b7efb981909a0c13076129f12550a38eb2a727f818028a70b5b37e47" exitCode=7
Jul 18 01:11:08.046607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:08.046202    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" event={"ID":"e1f27b62-645f-4387-ab8a-4833d3baa022","Type":"ContainerDied","Data":"a46669a2b7efb981909a0c13076129f12550a38eb2a727f818028a70b5b37e47"}
Jul 18 01:11:08.046607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:08.046355    2554 scope.go:117] "RemoveContainer" containerID="a46669a2b7efb981909a0c13076129f12550a38eb2a727f818028a70b5b37e47"
Jul 18 01:11:09.054874 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:09.054644    2554 generic.go:356] "Generic (PLEG): container finished" podID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerID="71c3f93d33629d9c066f1cedce046d2487e5b33ba14b46d5bc6d8a3726789d4c" exitCode=7
Jul 18 01:11:09.054874 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:09.054704    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" event={"ID":"e1f27b62-645f-4387-ab8a-4833d3baa022","Type":"ContainerDied","Data":"71c3f93d33629d9c066f1cedce046d2487e5b33ba14b46d5bc6d8a3726789d4c"}
Jul 18 01:11:09.054874 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:09.054751    2554 scope.go:117] "RemoveContainer" containerID="a46669a2b7efb981909a0c13076129f12550a38eb2a727f818028a70b5b37e47"
Jul 18 01:11:09.055403 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:09.054915    2554 scope.go:117] "RemoveContainer" containerID="71c3f93d33629d9c066f1cedce046d2487e5b33ba14b46d5bc6d8a3726789d4c"
Jul 18 01:11:09.055403 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:11:09.055198    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7_odh-ai-gateway-infra(e1f27b62-645f-4387-ab8a-4833d3baa022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022"
Jul 18 01:11:10.061696 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:10.061600    2554 scope.go:117] "RemoveContainer" containerID="71c3f93d33629d9c066f1cedce046d2487e5b33ba14b46d5bc6d8a3726789d4c"
Jul 18 01:11:10.062051 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:11:10.061881    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7_odh-ai-gateway-infra(e1f27b62-645f-4387-ab8a-4833d3baa022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022"
Jul 18 01:11:13.078417 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:13.078381    2554 generic.go:356] "Generic (PLEG): container finished" podID="27e9a9e2-9034-4481-a00c-31990de5d687" containerID="17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87" exitCode=0
Jul 18 01:11:13.078932 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:13.078454    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" event={"ID":"27e9a9e2-9034-4481-a00c-31990de5d687","Type":"ContainerDied","Data":"17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87"}
Jul 18 01:11:13.078932 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:13.078886    2554 scope.go:117] "RemoveContainer" containerID="17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87"
Jul 18 01:11:13.079908 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:13.079881    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:11:13.080003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:13.079945    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:11:13.080003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:13.079979    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:11:14.084933 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:14.084897    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" event={"ID":"27e9a9e2-9034-4481-a00c-31990de5d687","Type":"ContainerStarted","Data":"a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3"}
Jul 18 01:11:14.930819 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:14.930776    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:14.931991 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:14.931962    2554 patch_prober.go:28] interesting pod/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 18 01:11:14.932143 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:14.932014    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 18 01:11:15.931050 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:15.931002    2554 patch_prober.go:28] interesting pod/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 18 01:11:15.931545 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:15.931093    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 18 01:11:16.931292 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:16.931248    2554 patch_prober.go:28] interesting pod/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 18 01:11:16.931803 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:16.931317    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 18 01:11:17.931218 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:17.931169    2554 patch_prober.go:28] interesting pod/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 18 01:11:17.931428 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:17.931235    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 18 01:11:18.016673 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:18.016604    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:18.934398 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:18.934366    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:18.935429 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:18.935409    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:11:24.337437 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:24.337398    2554 scope.go:117] "RemoveContainer" containerID="71c3f93d33629d9c066f1cedce046d2487e5b33ba14b46d5bc6d8a3726789d4c"
Jul 18 01:11:25.146814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:25.146784    2554 generic.go:356] "Generic (PLEG): container finished" podID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerID="23b8e6f33d326f564f71b5dff196b5fbb713cdc58bbea7a4a7cc6bb16a66e534" exitCode=7
Jul 18 01:11:25.146980 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:25.146829    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" event={"ID":"e1f27b62-645f-4387-ab8a-4833d3baa022","Type":"ContainerDied","Data":"23b8e6f33d326f564f71b5dff196b5fbb713cdc58bbea7a4a7cc6bb16a66e534"}
Jul 18 01:11:25.146980 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:25.146877    2554 scope.go:117] "RemoveContainer" containerID="71c3f93d33629d9c066f1cedce046d2487e5b33ba14b46d5bc6d8a3726789d4c"
Jul 18 01:11:25.147243 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:25.147221    2554 scope.go:117] "RemoveContainer" containerID="23b8e6f33d326f564f71b5dff196b5fbb713cdc58bbea7a4a7cc6bb16a66e534"
Jul 18 01:11:25.147490 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:11:25.147461    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7_odh-ai-gateway-infra(e1f27b62-645f-4387-ab8a-4833d3baa022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022"
Jul 18 01:11:26.171042 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:26.171006    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"]
Jul 18 01:11:26.308560 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:26.308533    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"
Jul 18 01:11:26.391301 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:26.391260    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f27b62-645f-4387-ab8a-4833d3baa022-maas-api-service-ca\") pod \"e1f27b62-645f-4387-ab8a-4833d3baa022\" (UID: \"e1f27b62-645f-4387-ab8a-4833d3baa022\") "
Jul 18 01:11:26.391667 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:26.391608    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f27b62-645f-4387-ab8a-4833d3baa022-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1f27b62-645f-4387-ab8a-4833d3baa022" (UID: "e1f27b62-645f-4387-ab8a-4833d3baa022"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:11:26.492277 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:26.492243    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f27b62-645f-4387-ab8a-4833d3baa022-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:11:27.159872 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:27.159835    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7" event={"ID":"e1f27b62-645f-4387-ab8a-4833d3baa022","Type":"ContainerDied","Data":"5dcf61791f3111f0b8d9c9ada0f83fe90c4d7672d7504c05d79301f5f8459ff8"}
Jul 18 01:11:27.160048 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:27.159893    2554 scope.go:117] "RemoveContainer" containerID="23b8e6f33d326f564f71b5dff196b5fbb713cdc58bbea7a4a7cc6bb16a66e534"
Jul 18 01:11:27.160048 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:27.159846    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"
Jul 18 01:11:27.183186 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:27.183156    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"]
Jul 18 01:11:27.188397 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:27.188375    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-t5sl7"]
Jul 18 01:11:28.336198 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:28.336156    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" path="/var/lib/kubelet/pods/e1f27b62-645f-4387-ab8a-4833d3baa022/volumes"
Jul 18 01:11:38.363378 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.363341    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"]
Jul 18 01:11:38.363794 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.363760    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:11:38.363794 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.363773    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:11:38.363794 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.363784    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:11:38.363794 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.363790    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:11:38.363926 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.363872    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:11:38.363926 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.363884    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:11:38.366587 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.366569    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"
Jul 18 01:11:38.368913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.368888    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:11:38.374225 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.374198    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"]
Jul 18 01:11:38.511812 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.511773    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a194e-bcca-482d-9877-f701e3ff8114-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m\" (UID: \"f79a194e-bcca-482d-9877-f701e3ff8114\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"
Jul 18 01:11:38.613028 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.612986    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a194e-bcca-482d-9877-f701e3ff8114-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m\" (UID: \"f79a194e-bcca-482d-9877-f701e3ff8114\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"
Jul 18 01:11:38.613668 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.613589    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a194e-bcca-482d-9877-f701e3ff8114-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m\" (UID: \"f79a194e-bcca-482d-9877-f701e3ff8114\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"
Jul 18 01:11:38.679693 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.679660    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"
Jul 18 01:11:38.809787 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:38.809751    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"]
Jul 18 01:11:38.812212 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:11:38.812184    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79a194e_bcca_482d_9877_f701e3ff8114.slice/crio-9dc3bc1f75de2aa417b520b32fdd9a6c576aac33ff6a1c3f9aef37d3294292a3 WatchSource:0}: Error finding container 9dc3bc1f75de2aa417b520b32fdd9a6c576aac33ff6a1c3f9aef37d3294292a3: Status 404 returned error can't find the container with id 9dc3bc1f75de2aa417b520b32fdd9a6c576aac33ff6a1c3f9aef37d3294292a3
Jul 18 01:11:39.220230 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:39.220130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" event={"ID":"f79a194e-bcca-482d-9877-f701e3ff8114","Type":"ContainerStarted","Data":"da64196b07925aaffa4a0811982fe4276ad9f5c7810fcb9754e30d60fb0bae1e"}
Jul 18 01:11:39.220230 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:39.220195    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" event={"ID":"f79a194e-bcca-482d-9877-f701e3ff8114","Type":"ContainerStarted","Data":"9dc3bc1f75de2aa417b520b32fdd9a6c576aac33ff6a1c3f9aef37d3294292a3"}
Jul 18 01:11:39.235907 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:39.235848    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" podStartSLOduration=1.23583159 podStartE2EDuration="1.23583159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:11:39.234361304 +0000 UTC m=+1855.360296297" watchObservedRunningTime="2026-07-18 01:11:39.23583159 +0000 UTC m=+1855.361766569"
Jul 18 01:11:40.227743 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:40.227700    2554 generic.go:356] "Generic (PLEG): container finished" podID="f79a194e-bcca-482d-9877-f701e3ff8114" containerID="da64196b07925aaffa4a0811982fe4276ad9f5c7810fcb9754e30d60fb0bae1e" exitCode=6
Jul 18 01:11:40.227743 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:40.227739    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" event={"ID":"f79a194e-bcca-482d-9877-f701e3ff8114","Type":"ContainerDied","Data":"da64196b07925aaffa4a0811982fe4276ad9f5c7810fcb9754e30d60fb0bae1e"}
Jul 18 01:11:40.228304 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:40.227993    2554 scope.go:117] "RemoveContainer" containerID="da64196b07925aaffa4a0811982fe4276ad9f5c7810fcb9754e30d60fb0bae1e"
Jul 18 01:11:40.815273 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:11:40.815239    2554 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79a194e_bcca_482d_9877_f701e3ff8114.slice/crio-d4d76f621ba35e0e27f90b54acc3b35c4e3add0e4dec2f3ca3aa47b2f866467f.scope\": RecentStats: unable to find data in memory cache]"
Jul 18 01:11:41.234416 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:41.234331    2554 generic.go:356] "Generic (PLEG): container finished" podID="f79a194e-bcca-482d-9877-f701e3ff8114" containerID="d4d76f621ba35e0e27f90b54acc3b35c4e3add0e4dec2f3ca3aa47b2f866467f" exitCode=6
Jul 18 01:11:41.234825 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:41.234417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" event={"ID":"f79a194e-bcca-482d-9877-f701e3ff8114","Type":"ContainerDied","Data":"d4d76f621ba35e0e27f90b54acc3b35c4e3add0e4dec2f3ca3aa47b2f866467f"}
Jul 18 01:11:41.234825 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:41.234476    2554 scope.go:117] "RemoveContainer" containerID="da64196b07925aaffa4a0811982fe4276ad9f5c7810fcb9754e30d60fb0bae1e"
Jul 18 01:11:41.234825 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:41.234671    2554 scope.go:117] "RemoveContainer" containerID="d4d76f621ba35e0e27f90b54acc3b35c4e3add0e4dec2f3ca3aa47b2f866467f"
Jul 18 01:11:41.234933 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:11:41.234896    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m_odh-ai-gateway-infra(f79a194e-bcca-482d-9877-f701e3ff8114)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" podUID="f79a194e-bcca-482d-9877-f701e3ff8114"
Jul 18 01:11:42.243385 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:42.243359    2554 scope.go:117] "RemoveContainer" containerID="d4d76f621ba35e0e27f90b54acc3b35c4e3add0e4dec2f3ca3aa47b2f866467f"
Jul 18 01:11:42.243775 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:11:42.243565    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m_odh-ai-gateway-infra(f79a194e-bcca-482d-9877-f701e3ff8114)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" podUID="f79a194e-bcca-482d-9877-f701e3ff8114"
Jul 18 01:11:54.333026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:54.332994    2554 scope.go:117] "RemoveContainer" containerID="d4d76f621ba35e0e27f90b54acc3b35c4e3add0e4dec2f3ca3aa47b2f866467f"
Jul 18 01:11:55.308552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:55.308515    2554 generic.go:356] "Generic (PLEG): container finished" podID="f79a194e-bcca-482d-9877-f701e3ff8114" containerID="593e4c0a0b13cc0a2e98462e7ea564a96a4c5a6569d8e66f0d268ac778468c7b" exitCode=6
Jul 18 01:11:55.308744 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:55.308568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" event={"ID":"f79a194e-bcca-482d-9877-f701e3ff8114","Type":"ContainerDied","Data":"593e4c0a0b13cc0a2e98462e7ea564a96a4c5a6569d8e66f0d268ac778468c7b"}
Jul 18 01:11:55.308744 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:55.308638    2554 scope.go:117] "RemoveContainer" containerID="d4d76f621ba35e0e27f90b54acc3b35c4e3add0e4dec2f3ca3aa47b2f866467f"
Jul 18 01:11:55.308967 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:55.308950    2554 scope.go:117] "RemoveContainer" containerID="593e4c0a0b13cc0a2e98462e7ea564a96a4c5a6569d8e66f0d268ac778468c7b"
Jul 18 01:11:55.309180 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:11:55.309157    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m_odh-ai-gateway-infra(f79a194e-bcca-482d-9877-f701e3ff8114)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" podUID="f79a194e-bcca-482d-9877-f701e3ff8114"
Jul 18 01:11:56.337707 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:56.337678    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"]
Jul 18 01:11:56.470500 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:56.470476    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"
Jul 18 01:11:56.488176 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:56.488145    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a194e-bcca-482d-9877-f701e3ff8114-maas-api-service-ca\") pod \"f79a194e-bcca-482d-9877-f701e3ff8114\" (UID: \"f79a194e-bcca-482d-9877-f701e3ff8114\") "
Jul 18 01:11:56.488507 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:56.488479    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79a194e-bcca-482d-9877-f701e3ff8114-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f79a194e-bcca-482d-9877-f701e3ff8114" (UID: "f79a194e-bcca-482d-9877-f701e3ff8114"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:11:56.488652 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:56.488608    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a194e-bcca-482d-9877-f701e3ff8114-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:11:57.320864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:57.320829    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"
Jul 18 01:11:57.320864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:57.320858    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m" event={"ID":"f79a194e-bcca-482d-9877-f701e3ff8114","Type":"ContainerDied","Data":"9dc3bc1f75de2aa417b520b32fdd9a6c576aac33ff6a1c3f9aef37d3294292a3"}
Jul 18 01:11:57.321104 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:57.320918    2554 scope.go:117] "RemoveContainer" containerID="593e4c0a0b13cc0a2e98462e7ea564a96a4c5a6569d8e66f0d268ac778468c7b"
Jul 18 01:11:57.340900 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:57.340874    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"]
Jul 18 01:11:57.351444 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:57.348836    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bcw9m"]
Jul 18 01:11:58.336486 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:11:58.336451    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" path="/var/lib/kubelet/pods/f79a194e-bcca-482d-9877-f701e3ff8114/volumes"
Jul 18 01:12:06.415799 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.415768    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"]
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416193    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416205    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416213    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416219    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416226    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416235    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416246    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416252    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416318    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f27b62-645f-4387-ab8a-4833d3baa022" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416331    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416338    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.416408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.416344    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79a194e-bcca-482d-9877-f701e3ff8114" containerName="revoke-keys"
Jul 18 01:12:06.418795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.418778    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"
Jul 18 01:12:06.421193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.421168    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:12:06.427013 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.426987    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"]
Jul 18 01:12:06.477905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.477864    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b195c3-0bae-4c3b-82b6-4246a7d981d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8\" (UID: \"74b195c3-0bae-4c3b-82b6-4246a7d981d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"
Jul 18 01:12:06.485440 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.485405    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"]
Jul 18 01:12:06.485720 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.485670    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy" containerID="cri-o://a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3" gracePeriod=30
Jul 18 01:12:06.578897 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.578852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b195c3-0bae-4c3b-82b6-4246a7d981d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8\" (UID: \"74b195c3-0bae-4c3b-82b6-4246a7d981d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"
Jul 18 01:12:06.579500 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.579469    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b195c3-0bae-4c3b-82b6-4246a7d981d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8\" (UID: \"74b195c3-0bae-4c3b-82b6-4246a7d981d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"
Jul 18 01:12:06.731120 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:06.731035    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"
Jul 18 01:12:07.067785 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:07.067759    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"]
Jul 18 01:12:07.070185 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:12:07.070151    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b195c3_0bae_4c3b_82b6_4246a7d981d3.slice/crio-29dfd28b93d59fda4738c208429616e08cc1abc03679feb3123a9c58730e1ef4 WatchSource:0}: Error finding container 29dfd28b93d59fda4738c208429616e08cc1abc03679feb3123a9c58730e1ef4: Status 404 returned error can't find the container with id 29dfd28b93d59fda4738c208429616e08cc1abc03679feb3123a9c58730e1ef4
Jul 18 01:12:07.371841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:07.371801    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" event={"ID":"74b195c3-0bae-4c3b-82b6-4246a7d981d3","Type":"ContainerStarted","Data":"91ed272db5d63ee7b250b384e5c2b7d6254a246a1e047ec130c839b8bf4b1fb1"}
Jul 18 01:12:07.372013 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:07.371864    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" event={"ID":"74b195c3-0bae-4c3b-82b6-4246a7d981d3","Type":"ContainerStarted","Data":"29dfd28b93d59fda4738c208429616e08cc1abc03679feb3123a9c58730e1ef4"}
Jul 18 01:12:07.388811 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:07.388760    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" podStartSLOduration=1.388743634 podStartE2EDuration="1.388743634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:12:07.386315739 +0000 UTC m=+1883.512250742" watchObservedRunningTime="2026-07-18 01:12:07.388743634 +0000 UTC m=+1883.514678613"
Jul 18 01:12:08.377918 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:08.377885    2554 generic.go:356] "Generic (PLEG): container finished" podID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerID="91ed272db5d63ee7b250b384e5c2b7d6254a246a1e047ec130c839b8bf4b1fb1" exitCode=7
Jul 18 01:12:08.378305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:08.377964    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" event={"ID":"74b195c3-0bae-4c3b-82b6-4246a7d981d3","Type":"ContainerDied","Data":"91ed272db5d63ee7b250b384e5c2b7d6254a246a1e047ec130c839b8bf4b1fb1"}
Jul 18 01:12:08.378305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:08.378172    2554 scope.go:117] "RemoveContainer" containerID="91ed272db5d63ee7b250b384e5c2b7d6254a246a1e047ec130c839b8bf4b1fb1"
Jul 18 01:12:09.385003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:09.384967    2554 generic.go:356] "Generic (PLEG): container finished" podID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerID="03303d78bd3d379b565ae2f0c7e4a0fe72b4dedec38ee1b75f598bd34ba2ba87" exitCode=7
Jul 18 01:12:09.385462 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:09.385016    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" event={"ID":"74b195c3-0bae-4c3b-82b6-4246a7d981d3","Type":"ContainerDied","Data":"03303d78bd3d379b565ae2f0c7e4a0fe72b4dedec38ee1b75f598bd34ba2ba87"}
Jul 18 01:12:09.385462 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:09.385075    2554 scope.go:117] "RemoveContainer" containerID="91ed272db5d63ee7b250b384e5c2b7d6254a246a1e047ec130c839b8bf4b1fb1"
Jul 18 01:12:09.385462 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:09.385318    2554 scope.go:117] "RemoveContainer" containerID="03303d78bd3d379b565ae2f0c7e4a0fe72b4dedec38ee1b75f598bd34ba2ba87"
Jul 18 01:12:09.385587 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:09.385567    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8_odh-ai-gateway-infra(74b195c3-0bae-4c3b-82b6-4246a7d981d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3"
Jul 18 01:12:10.391338 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:10.391312    2554 scope.go:117] "RemoveContainer" containerID="03303d78bd3d379b565ae2f0c7e4a0fe72b4dedec38ee1b75f598bd34ba2ba87"
Jul 18 01:12:10.391739 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:10.391555    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8_odh-ai-gateway-infra(74b195c3-0bae-4c3b-82b6-4246a7d981d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3"
Jul 18 01:12:11.745869 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.745842    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:12:11.833504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833475    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e9a9e2-9034-4481-a00c-31990de5d687-istiod-ca-cert\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.833651 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833515    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlxxg\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-kube-api-access-zlxxg\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.833651 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833564    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e9a9e2-9034-4481-a00c-31990de5d687-istio-podinfo\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.833651 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833604    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-credential-socket\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.833831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833720    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-envoy\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.833831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833743    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-certs\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.833831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833769    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-socket\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.833831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833820    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-data\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.834023 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833865    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-istio-token\") pod \"27e9a9e2-9034-4481-a00c-31990de5d687\" (UID: \"27e9a9e2-9034-4481-a00c-31990de5d687\") "
Jul 18 01:12:11.834023 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.833897    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e9a9e2-9034-4481-a00c-31990de5d687-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:12:11.834138 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.834007    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:12:11.834224 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.834198    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:12:11.834287 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.834246    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.834287 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.834267    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e9a9e2-9034-4481-a00c-31990de5d687-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.834390 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.834336    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:12:11.834831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.834795    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-data" (OuterVolumeSpecName: "istio-data") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:12:11.836016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.835988    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-kube-api-access-zlxxg" (OuterVolumeSpecName: "kube-api-access-zlxxg") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "kube-api-access-zlxxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:12:11.836193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.836168    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:12:11.836358 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.836335    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-istio-token" (OuterVolumeSpecName: "istio-token") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:12:11.836358 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.836341    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27e9a9e2-9034-4481-a00c-31990de5d687-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "27e9a9e2-9034-4481-a00c-31990de5d687" (UID: "27e9a9e2-9034-4481-a00c-31990de5d687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:12:11.935491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.935446    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxxg\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-kube-api-access-zlxxg\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.935491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.935487    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e9a9e2-9034-4481-a00c-31990de5d687-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.935491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.935498    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.935752 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.935509    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.935752 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.935535    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.935752 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.935544    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e9a9e2-9034-4481-a00c-31990de5d687-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:11.935752 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:11.935552    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e9a9e2-9034-4481-a00c-31990de5d687-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:12.403173 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.403140    2554 generic.go:356] "Generic (PLEG): container finished" podID="27e9a9e2-9034-4481-a00c-31990de5d687" containerID="a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3" exitCode=0
Jul 18 01:12:12.403340 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.403183    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"
Jul 18 01:12:12.403340 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.403216    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" event={"ID":"27e9a9e2-9034-4481-a00c-31990de5d687","Type":"ContainerDied","Data":"a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3"}
Jul 18 01:12:12.403340 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.403268    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw" event={"ID":"27e9a9e2-9034-4481-a00c-31990de5d687","Type":"ContainerDied","Data":"934b1111f0d4705d3cb051164bffb9967dda9a6d47a6b5b49aae7facdfa5c883"}
Jul 18 01:12:12.403340 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.403289    2554 scope.go:117] "RemoveContainer" containerID="a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3"
Jul 18 01:12:12.412236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.412219    2554 scope.go:117] "RemoveContainer" containerID="17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87"
Jul 18 01:12:12.420689 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.420669    2554 scope.go:117] "RemoveContainer" containerID="a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3"
Jul 18 01:12:12.420954 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:12.420932    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3\": container with ID starting with a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3 not found: ID does not exist" containerID="a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3"
Jul 18 01:12:12.421009 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.420962    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3"} err="failed to get container status \"a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3\": rpc error: code = NotFound desc = could not find container \"a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3\": container with ID starting with a56cf5a61e18a92cf65af64996b427e31ce6c2b503acdaefbce1cf6fb07415f3 not found: ID does not exist"
Jul 18 01:12:12.421009 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.420981    2554 scope.go:117] "RemoveContainer" containerID="17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87"
Jul 18 01:12:12.421256 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:12.421238    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87\": container with ID starting with 17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87 not found: ID does not exist" containerID="17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87"
Jul 18 01:12:12.421310 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.421262    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87"} err="failed to get container status \"17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87\": rpc error: code = NotFound desc = could not find container \"17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87\": container with ID starting with 17f3e9f9073a5b5a6c5e5cd2833a529674c9b49430bd965707910b80f1dccc87 not found: ID does not exist"
Jul 18 01:12:12.422172 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.422153    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"]
Jul 18 01:12:12.425164 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:12.425140    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bf4f6c8d-gw-openshift-default-84d8f888d7-xzdrw"]
Jul 18 01:12:14.336707 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:14.336676    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" path="/var/lib/kubelet/pods/27e9a9e2-9034-4481-a00c-31990de5d687/volumes"
Jul 18 01:12:18.165401 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.165368    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"]
Jul 18 01:12:18.165928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.165824    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy"
Jul 18 01:12:18.165928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.165838    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy"
Jul 18 01:12:18.165928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.165855    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy"
Jul 18 01:12:18.165928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.165861    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy"
Jul 18 01:12:18.165928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.165931    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy"
Jul 18 01:12:18.167786 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.167766    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.170380 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.170360    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4701d191-openshift-default-dockercfg-k7d65\""
Jul 18 01:12:18.182560 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.182535    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"]
Jul 18 01:12:18.296373 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-credential-socket\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-data\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296404    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-certs\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296492    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-istio-token\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296530    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1522746-3999-46cf-8cb9-1704658f2992-istiod-ca-cert\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxbbv\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-kube-api-access-gxbbv\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-envoy\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296660    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-socket\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.296764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.296733    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1522746-3999-46cf-8cb9-1704658f2992-istio-podinfo\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397433 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397403    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxbbv\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-kube-api-access-gxbbv\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-envoy\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397476    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-socket\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1522746-3999-46cf-8cb9-1704658f2992-istio-podinfo\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397734 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-credential-socket\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397734 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397610    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-data\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397734 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397653    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-certs\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397734 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397709    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-istio-token\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.397909 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.397742    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1522746-3999-46cf-8cb9-1704658f2992-istiod-ca-cert\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.398112 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.398077    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-socket\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.398234 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.398129    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-credential-socket\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.398234 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.398187    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-certs\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.398370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.398242    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-data\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.398498 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.398478    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1522746-3999-46cf-8cb9-1704658f2992-istiod-ca-cert\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.399727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.399701    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-envoy\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.400140 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.400117    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1522746-3999-46cf-8cb9-1704658f2992-istio-podinfo\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.404843 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.404824    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-istio-token\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.405238 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.405218    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxbbv\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-kube-api-access-gxbbv\") pod \"e2e-ait-4701d191-openshift-default-6b94dc8697-sb465\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") " pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.482433 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.482352    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:18.623597 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.623569    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"]
Jul 18 01:12:18.625333 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:12:18.625301    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1522746_3999_46cf_8cb9_1704658f2992.slice/crio-8911a2f7f4e102897c5fffeb3beb5d0acfee4be510e78404a2629e0da8c15971 WatchSource:0}: Error finding container 8911a2f7f4e102897c5fffeb3beb5d0acfee4be510e78404a2629e0da8c15971: Status 404 returned error can't find the container with id 8911a2f7f4e102897c5fffeb3beb5d0acfee4be510e78404a2629e0da8c15971
Jul 18 01:12:18.627389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.627360    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:12:18.627510 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.627427    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:12:18.627510 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:18.627489    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:12:19.438393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:19.438357    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" event={"ID":"a1522746-3999-46cf-8cb9-1704658f2992","Type":"ContainerStarted","Data":"6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9"}
Jul 18 01:12:19.438844 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:19.438419    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" event={"ID":"a1522746-3999-46cf-8cb9-1704658f2992","Type":"ContainerStarted","Data":"8911a2f7f4e102897c5fffeb3beb5d0acfee4be510e78404a2629e0da8c15971"}
Jul 18 01:12:19.463046 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:19.462994    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" podStartSLOduration=1.462978209 podStartE2EDuration="1.462978209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:12:19.46075164 +0000 UTC m=+1895.586686622" watchObservedRunningTime="2026-07-18 01:12:19.462978209 +0000 UTC m=+1895.588913188"
Jul 18 01:12:19.483232 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:19.483199    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:19.487921 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:19.487897    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:20.443810 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:20.443780    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:20.444688 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:20.444669    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:24.336742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.336713    2554 scope.go:117] "RemoveContainer" containerID="03303d78bd3d379b565ae2f0c7e4a0fe72b4dedec38ee1b75f598bd34ba2ba87"
Jul 18 01:12:24.430876 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.430833    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"]
Jul 18 01:12:24.431572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.431546    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e9a9e2-9034-4481-a00c-31990de5d687" containerName="istio-proxy"
Jul 18 01:12:24.434830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.434812    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"
Jul 18 01:12:24.443173 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.443117    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"]
Jul 18 01:12:24.560700 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.560665    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd329242-82f2-4844-ba11-7c21d89df8fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-742q8\" (UID: \"fd329242-82f2-4844-ba11-7c21d89df8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"
Jul 18 01:12:24.661995 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.661970    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd329242-82f2-4844-ba11-7c21d89df8fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-742q8\" (UID: \"fd329242-82f2-4844-ba11-7c21d89df8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"
Jul 18 01:12:24.662651 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.662602    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd329242-82f2-4844-ba11-7c21d89df8fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-742q8\" (UID: \"fd329242-82f2-4844-ba11-7c21d89df8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"
Jul 18 01:12:24.748036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.747998    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"
Jul 18 01:12:24.930704 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:24.929290    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"]
Jul 18 01:12:24.933504 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:12:24.933465    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd329242_82f2_4844_ba11_7c21d89df8fc.slice/crio-8b8449428461a469743a3c2783ed3fdc2649086a5d4c7f141b84f10df61cedd6 WatchSource:0}: Error finding container 8b8449428461a469743a3c2783ed3fdc2649086a5d4c7f141b84f10df61cedd6: Status 404 returned error can't find the container with id 8b8449428461a469743a3c2783ed3fdc2649086a5d4c7f141b84f10df61cedd6
Jul 18 01:12:25.469033 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:25.468929    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" event={"ID":"fd329242-82f2-4844-ba11-7c21d89df8fc","Type":"ContainerStarted","Data":"54a6bafb59e5f846c2dfdd9c3ca4cf3829af9b9701ea9ec846090521ed36c0dd"}
Jul 18 01:12:25.469033 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:25.468996    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" event={"ID":"fd329242-82f2-4844-ba11-7c21d89df8fc","Type":"ContainerStarted","Data":"8b8449428461a469743a3c2783ed3fdc2649086a5d4c7f141b84f10df61cedd6"}
Jul 18 01:12:25.470765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:25.470739    2554 generic.go:356] "Generic (PLEG): container finished" podID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerID="341ef46b105a6023de7590709c76916ad81febd3b8d5396c98ac442a5f5ff67b" exitCode=6
Jul 18 01:12:25.470882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:25.470771    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" event={"ID":"74b195c3-0bae-4c3b-82b6-4246a7d981d3","Type":"ContainerDied","Data":"341ef46b105a6023de7590709c76916ad81febd3b8d5396c98ac442a5f5ff67b"}
Jul 18 01:12:25.470882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:25.470806    2554 scope.go:117] "RemoveContainer" containerID="03303d78bd3d379b565ae2f0c7e4a0fe72b4dedec38ee1b75f598bd34ba2ba87"
Jul 18 01:12:25.471084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:25.471065    2554 scope.go:117] "RemoveContainer" containerID="341ef46b105a6023de7590709c76916ad81febd3b8d5396c98ac442a5f5ff67b"
Jul 18 01:12:25.471265 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:25.471247    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8_odh-ai-gateway-infra(74b195c3-0bae-4c3b-82b6-4246a7d981d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3"
Jul 18 01:12:25.484495 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:25.484454    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" podStartSLOduration=1.484440036 podStartE2EDuration="1.484440036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:12:25.483936108 +0000 UTC m=+1901.609871088" watchObservedRunningTime="2026-07-18 01:12:25.484440036 +0000 UTC m=+1901.610375015"
Jul 18 01:12:26.481219 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.481185    2554 generic.go:356] "Generic (PLEG): container finished" podID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerID="54a6bafb59e5f846c2dfdd9c3ca4cf3829af9b9701ea9ec846090521ed36c0dd" exitCode=7
Jul 18 01:12:26.481635 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.481237    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" event={"ID":"fd329242-82f2-4844-ba11-7c21d89df8fc","Type":"ContainerDied","Data":"54a6bafb59e5f846c2dfdd9c3ca4cf3829af9b9701ea9ec846090521ed36c0dd"}
Jul 18 01:12:26.481635 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.481464    2554 scope.go:117] "RemoveContainer" containerID="54a6bafb59e5f846c2dfdd9c3ca4cf3829af9b9701ea9ec846090521ed36c0dd"
Jul 18 01:12:26.514664 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.514242    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"]
Jul 18 01:12:26.679093 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.679067    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"
Jul 18 01:12:26.782587 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.782560    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b195c3-0bae-4c3b-82b6-4246a7d981d3-maas-api-service-ca\") pod \"74b195c3-0bae-4c3b-82b6-4246a7d981d3\" (UID: \"74b195c3-0bae-4c3b-82b6-4246a7d981d3\") "
Jul 18 01:12:26.782925 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.782902    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b195c3-0bae-4c3b-82b6-4246a7d981d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74b195c3-0bae-4c3b-82b6-4246a7d981d3" (UID: "74b195c3-0bae-4c3b-82b6-4246a7d981d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:12:26.883315 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:26.883288    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b195c3-0bae-4c3b-82b6-4246a7d981d3-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:27.488913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.488879    2554 generic.go:356] "Generic (PLEG): container finished" podID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerID="81e4f6df098eeeb6cb16cdc4a313d2b842e6db1ee7a4d9b5fb03bd990e4ccdd9" exitCode=7
Jul 18 01:12:27.489314 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.488968    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" event={"ID":"fd329242-82f2-4844-ba11-7c21d89df8fc","Type":"ContainerDied","Data":"81e4f6df098eeeb6cb16cdc4a313d2b842e6db1ee7a4d9b5fb03bd990e4ccdd9"}
Jul 18 01:12:27.489314 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.489029    2554 scope.go:117] "RemoveContainer" containerID="54a6bafb59e5f846c2dfdd9c3ca4cf3829af9b9701ea9ec846090521ed36c0dd"
Jul 18 01:12:27.489314 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.489214    2554 scope.go:117] "RemoveContainer" containerID="81e4f6df098eeeb6cb16cdc4a313d2b842e6db1ee7a4d9b5fb03bd990e4ccdd9"
Jul 18 01:12:27.489470 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:27.489415    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-742q8_odh-ai-gateway-infra(fd329242-82f2-4844-ba11-7c21d89df8fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc"
Jul 18 01:12:27.490381 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.490364    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"
Jul 18 01:12:27.490460 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.490389    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8" event={"ID":"74b195c3-0bae-4c3b-82b6-4246a7d981d3","Type":"ContainerDied","Data":"29dfd28b93d59fda4738c208429616e08cc1abc03679feb3123a9c58730e1ef4"}
Jul 18 01:12:27.500997 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.500976    2554 scope.go:117] "RemoveContainer" containerID="341ef46b105a6023de7590709c76916ad81febd3b8d5396c98ac442a5f5ff67b"
Jul 18 01:12:27.517740 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.517715    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"]
Jul 18 01:12:27.523735 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:27.523711    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-qxdw8"]
Jul 18 01:12:28.337820 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.337783    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" path="/var/lib/kubelet/pods/74b195c3-0bae-4c3b-82b6-4246a7d981d3/volumes"
Jul 18 01:12:28.404218 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404183    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"]
Jul 18 01:12:28.404749 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404733    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404752    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404768    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404776    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404790    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404797    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404976 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404878    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404976 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404890    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.404976 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.404900    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b195c3-0bae-4c3b-82b6-4246a7d981d3" containerName="revoke-keys"
Jul 18 01:12:28.407738 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.407716    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"
Jul 18 01:12:28.413537 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.413512    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"]
Jul 18 01:12:28.498318 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.498292    2554 scope.go:117] "RemoveContainer" containerID="81e4f6df098eeeb6cb16cdc4a313d2b842e6db1ee7a4d9b5fb03bd990e4ccdd9"
Jul 18 01:12:28.498788 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:28.498531    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-742q8_odh-ai-gateway-infra(fd329242-82f2-4844-ba11-7c21d89df8fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc"
Jul 18 01:12:28.499686 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.499659    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fdca88-cd94-409a-90a9-2de141ca6122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-h864w\" (UID: \"69fdca88-cd94-409a-90a9-2de141ca6122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"
Jul 18 01:12:28.600851 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.600756    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fdca88-cd94-409a-90a9-2de141ca6122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-h864w\" (UID: \"69fdca88-cd94-409a-90a9-2de141ca6122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"
Jul 18 01:12:28.601389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.601368    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fdca88-cd94-409a-90a9-2de141ca6122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-h864w\" (UID: \"69fdca88-cd94-409a-90a9-2de141ca6122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"
Jul 18 01:12:28.721426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.721385    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"
Jul 18 01:12:28.862897 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:28.862869    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"]
Jul 18 01:12:28.865625 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:12:28.865585    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fdca88_cd94_409a_90a9_2de141ca6122.slice/crio-5956ab36a77134527b9f556195e744d40a227b9f43ae287e6e233312b995fc57 WatchSource:0}: Error finding container 5956ab36a77134527b9f556195e744d40a227b9f43ae287e6e233312b995fc57: Status 404 returned error can't find the container with id 5956ab36a77134527b9f556195e744d40a227b9f43ae287e6e233312b995fc57
Jul 18 01:12:29.506588 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:29.506559    2554 generic.go:356] "Generic (PLEG): container finished" podID="69fdca88-cd94-409a-90a9-2de141ca6122" containerID="bb4730dbffe355852163185cf679c278ab04ceb7a5f4c4982c3df36799bfb7b8" exitCode=6
Jul 18 01:12:29.506999 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:29.506662    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" event={"ID":"69fdca88-cd94-409a-90a9-2de141ca6122","Type":"ContainerDied","Data":"bb4730dbffe355852163185cf679c278ab04ceb7a5f4c4982c3df36799bfb7b8"}
Jul 18 01:12:29.506999 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:29.506712    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" event={"ID":"69fdca88-cd94-409a-90a9-2de141ca6122","Type":"ContainerStarted","Data":"5956ab36a77134527b9f556195e744d40a227b9f43ae287e6e233312b995fc57"}
Jul 18 01:12:29.506999 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:29.506897    2554 scope.go:117] "RemoveContainer" containerID="bb4730dbffe355852163185cf679c278ab04ceb7a5f4c4982c3df36799bfb7b8"
Jul 18 01:12:30.513067 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.513034    2554 generic.go:356] "Generic (PLEG): container finished" podID="a1522746-3999-46cf-8cb9-1704658f2992" containerID="6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9" exitCode=0
Jul 18 01:12:30.513586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.513112    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" event={"ID":"a1522746-3999-46cf-8cb9-1704658f2992","Type":"ContainerDied","Data":"6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9"}
Jul 18 01:12:30.513704 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.513685    2554 scope.go:117] "RemoveContainer" containerID="6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9"
Jul 18 01:12:30.514738 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.514708    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:12:30.514862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.514796    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:12:30.514862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.514843    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:12:30.515075 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.515054    2554 generic.go:356] "Generic (PLEG): container finished" podID="69fdca88-cd94-409a-90a9-2de141ca6122" containerID="294d32d3ca89af81a6ea8b11ffacbcdc74eaba43f76e0590480a4b8843478abd" exitCode=6
Jul 18 01:12:30.515151 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.515133    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" event={"ID":"69fdca88-cd94-409a-90a9-2de141ca6122","Type":"ContainerDied","Data":"294d32d3ca89af81a6ea8b11ffacbcdc74eaba43f76e0590480a4b8843478abd"}
Jul 18 01:12:30.515194 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.515184    2554 scope.go:117] "RemoveContainer" containerID="bb4730dbffe355852163185cf679c278ab04ceb7a5f4c4982c3df36799bfb7b8"
Jul 18 01:12:30.515310 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:30.515296    2554 scope.go:117] "RemoveContainer" containerID="294d32d3ca89af81a6ea8b11ffacbcdc74eaba43f76e0590480a4b8843478abd"
Jul 18 01:12:30.515516 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:30.515496    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-h864w_odh-ai-gateway-infra(69fdca88-cd94-409a-90a9-2de141ca6122)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" podUID="69fdca88-cd94-409a-90a9-2de141ca6122"
Jul 18 01:12:31.526499 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:31.526450    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" event={"ID":"a1522746-3999-46cf-8cb9-1704658f2992","Type":"ContainerStarted","Data":"6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268"}
Jul 18 01:12:31.530673 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:31.530649    2554 scope.go:117] "RemoveContainer" containerID="294d32d3ca89af81a6ea8b11ffacbcdc74eaba43f76e0590480a4b8843478abd"
Jul 18 01:12:31.531113 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:31.531075    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-h864w_odh-ai-gateway-infra(69fdca88-cd94-409a-90a9-2de141ca6122)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" podUID="69fdca88-cd94-409a-90a9-2de141ca6122"
Jul 18 01:12:32.483525 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:32.483481    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:32.484752 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:32.484726    2554 patch_prober.go:28] interesting pod/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 18 01:12:32.484872 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:32.484778    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 18 01:12:33.482984 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:33.482948    2554 patch_prober.go:28] interesting pod/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 18 01:12:33.483397 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:33.483008    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 18 01:12:34.483545 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:34.483506    2554 patch_prober.go:28] interesting pod/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 18 01:12:34.483959 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:34.483565    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 18 01:12:35.443781 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:35.443743    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:35.486608 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:35.486575    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:35.552694 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:35.552665    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:12:39.330886 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:39.330854    2554 scope.go:117] "RemoveContainer" containerID="81e4f6df098eeeb6cb16cdc4a313d2b842e6db1ee7a4d9b5fb03bd990e4ccdd9"
Jul 18 01:12:40.576968 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:40.576932    2554 generic.go:356] "Generic (PLEG): container finished" podID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerID="557bd98dd8ec2c3bdbec02e9c9e7cba718f5a5e1d5f6af2843833158f68b5228" exitCode=7
Jul 18 01:12:40.577439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:40.577002    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" event={"ID":"fd329242-82f2-4844-ba11-7c21d89df8fc","Type":"ContainerDied","Data":"557bd98dd8ec2c3bdbec02e9c9e7cba718f5a5e1d5f6af2843833158f68b5228"}
Jul 18 01:12:40.577439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:40.577058    2554 scope.go:117] "RemoveContainer" containerID="81e4f6df098eeeb6cb16cdc4a313d2b842e6db1ee7a4d9b5fb03bd990e4ccdd9"
Jul 18 01:12:40.577439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:40.577410    2554 scope.go:117] "RemoveContainer" containerID="557bd98dd8ec2c3bdbec02e9c9e7cba718f5a5e1d5f6af2843833158f68b5228"
Jul 18 01:12:40.577719 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:40.577698    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-742q8_odh-ai-gateway-infra(fd329242-82f2-4844-ba11-7c21d89df8fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc"
Jul 18 01:12:41.602803 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:41.602769    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"]
Jul 18 01:12:41.742384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:41.742359    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"
Jul 18 01:12:41.837365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:41.837330    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd329242-82f2-4844-ba11-7c21d89df8fc-maas-api-service-ca\") pod \"fd329242-82f2-4844-ba11-7c21d89df8fc\" (UID: \"fd329242-82f2-4844-ba11-7c21d89df8fc\") "
Jul 18 01:12:41.837703 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:41.837679    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd329242-82f2-4844-ba11-7c21d89df8fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd329242-82f2-4844-ba11-7c21d89df8fc" (UID: "fd329242-82f2-4844-ba11-7c21d89df8fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:12:41.938862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:41.938783    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd329242-82f2-4844-ba11-7c21d89df8fc-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:42.589766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:42.589730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8" event={"ID":"fd329242-82f2-4844-ba11-7c21d89df8fc","Type":"ContainerDied","Data":"8b8449428461a469743a3c2783ed3fdc2649086a5d4c7f141b84f10df61cedd6"}
Jul 18 01:12:42.589766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:42.589751    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"
Jul 18 01:12:42.589998 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:42.589795    2554 scope.go:117] "RemoveContainer" containerID="557bd98dd8ec2c3bdbec02e9c9e7cba718f5a5e1d5f6af2843833158f68b5228"
Jul 18 01:12:42.607210 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:42.607184    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"]
Jul 18 01:12:42.613135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:42.613108    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-742q8"]
Jul 18 01:12:43.331004 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:43.330966    2554 scope.go:117] "RemoveContainer" containerID="294d32d3ca89af81a6ea8b11ffacbcdc74eaba43f76e0590480a4b8843478abd"
Jul 18 01:12:43.596521 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:43.596450    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" event={"ID":"69fdca88-cd94-409a-90a9-2de141ca6122","Type":"ContainerStarted","Data":"6b111a842b25e3367206def527d2051ebe9209cab68f0d03a83b442b1b2f1202"}
Jul 18 01:12:43.613237 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:43.613189    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" podStartSLOduration=15.613173503 podStartE2EDuration="15.613173503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:12:43.611159288 +0000 UTC m=+1919.737094268" watchObservedRunningTime="2026-07-18 01:12:43.613173503 +0000 UTC m=+1919.739108513"
Jul 18 01:12:44.336259 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:44.336229    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" path="/var/lib/kubelet/pods/fd329242-82f2-4844-ba11-7c21d89df8fc/volumes"
Jul 18 01:12:44.603060 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:44.602975    2554 generic.go:356] "Generic (PLEG): container finished" podID="69fdca88-cd94-409a-90a9-2de141ca6122" containerID="6b111a842b25e3367206def527d2051ebe9209cab68f0d03a83b442b1b2f1202" exitCode=6
Jul 18 01:12:44.603060 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:44.603011    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" event={"ID":"69fdca88-cd94-409a-90a9-2de141ca6122","Type":"ContainerDied","Data":"6b111a842b25e3367206def527d2051ebe9209cab68f0d03a83b442b1b2f1202"}
Jul 18 01:12:44.603060 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:44.603058    2554 scope.go:117] "RemoveContainer" containerID="294d32d3ca89af81a6ea8b11ffacbcdc74eaba43f76e0590480a4b8843478abd"
Jul 18 01:12:44.603354 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:44.603330    2554 scope.go:117] "RemoveContainer" containerID="6b111a842b25e3367206def527d2051ebe9209cab68f0d03a83b442b1b2f1202"
Jul 18 01:12:44.603652 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:12:44.603607    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-h864w_odh-ai-gateway-infra(69fdca88-cd94-409a-90a9-2de141ca6122)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" podUID="69fdca88-cd94-409a-90a9-2de141ca6122"
Jul 18 01:12:44.633864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:44.633833    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"]
Jul 18 01:12:45.742157 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:45.742129    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"
Jul 18 01:12:45.776981 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:45.776953    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fdca88-cd94-409a-90a9-2de141ca6122-maas-api-service-ca\") pod \"69fdca88-cd94-409a-90a9-2de141ca6122\" (UID: \"69fdca88-cd94-409a-90a9-2de141ca6122\") "
Jul 18 01:12:45.777335 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:45.777312    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fdca88-cd94-409a-90a9-2de141ca6122-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69fdca88-cd94-409a-90a9-2de141ca6122" (UID: "69fdca88-cd94-409a-90a9-2de141ca6122"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:12:45.777402 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:45.777389    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fdca88-cd94-409a-90a9-2de141ca6122-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:12:46.614677 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:46.614568    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"
Jul 18 01:12:46.614677 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:46.614567    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w" event={"ID":"69fdca88-cd94-409a-90a9-2de141ca6122","Type":"ContainerDied","Data":"5956ab36a77134527b9f556195e744d40a227b9f43ae287e6e233312b995fc57"}
Jul 18 01:12:46.614893 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:46.614710    2554 scope.go:117] "RemoveContainer" containerID="6b111a842b25e3367206def527d2051ebe9209cab68f0d03a83b442b1b2f1202"
Jul 18 01:12:46.629822 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:46.629800    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"]
Jul 18 01:12:46.635521 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:46.635496    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-h864w"]
Jul 18 01:12:48.336288 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:12:48.336257    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" path="/var/lib/kubelet/pods/69fdca88-cd94-409a-90a9-2de141ca6122/volumes"
Jul 18 01:13:06.460952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.460917    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"]
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461327    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461338    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461348    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461353    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461361    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461367    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461378    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461384    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461391    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461397    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461411    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461416    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461475    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461484    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461490    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:06.461826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461497    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.461826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.461513    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fdca88-cd94-409a-90a9-2de141ca6122" containerName="revoke-keys"
Jul 18 01:13:06.464667 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.464650    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"
Jul 18 01:13:06.466966 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.466938    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:13:06.471130 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.471105    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"]
Jul 18 01:13:06.592679 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.592648    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4aaec-c28f-42a9-bc7e-31816e80bd72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx\" (UID: \"efd4aaec-c28f-42a9-bc7e-31816e80bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"
Jul 18 01:13:06.694161 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.694129    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4aaec-c28f-42a9-bc7e-31816e80bd72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx\" (UID: \"efd4aaec-c28f-42a9-bc7e-31816e80bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"
Jul 18 01:13:06.694771 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.694753    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4aaec-c28f-42a9-bc7e-31816e80bd72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx\" (UID: \"efd4aaec-c28f-42a9-bc7e-31816e80bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"
Jul 18 01:13:06.776316 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.776291    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"
Jul 18 01:13:06.908017 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:06.907991    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"]
Jul 18 01:13:06.909584 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:13:06.909547    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd4aaec_c28f_42a9_bc7e_31816e80bd72.slice/crio-705cb3a17f5bcfa6d15846886e907867d1cd626044240f220f9264aed3e75365 WatchSource:0}: Error finding container 705cb3a17f5bcfa6d15846886e907867d1cd626044240f220f9264aed3e75365: Status 404 returned error can't find the container with id 705cb3a17f5bcfa6d15846886e907867d1cd626044240f220f9264aed3e75365
Jul 18 01:13:07.717171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:07.717133    2554 generic.go:356] "Generic (PLEG): container finished" podID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerID="268c37161d739d4abece94dc39ddaba2989d82a083cbb271fa36de5cb13d40ab" exitCode=6
Jul 18 01:13:07.717591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:07.717232    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" event={"ID":"efd4aaec-c28f-42a9-bc7e-31816e80bd72","Type":"ContainerDied","Data":"268c37161d739d4abece94dc39ddaba2989d82a083cbb271fa36de5cb13d40ab"}
Jul 18 01:13:07.717591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:07.717298    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" event={"ID":"efd4aaec-c28f-42a9-bc7e-31816e80bd72","Type":"ContainerStarted","Data":"705cb3a17f5bcfa6d15846886e907867d1cd626044240f220f9264aed3e75365"}
Jul 18 01:13:07.717591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:07.717373    2554 scope.go:117] "RemoveContainer" containerID="268c37161d739d4abece94dc39ddaba2989d82a083cbb271fa36de5cb13d40ab"
Jul 18 01:13:08.724076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:08.723982    2554 generic.go:356] "Generic (PLEG): container finished" podID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerID="0ab7fbb0df9faa71bce4f943a9c70a24fd146f7710b82f190ba285737a00416b" exitCode=6
Jul 18 01:13:08.724076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:08.724047    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" event={"ID":"efd4aaec-c28f-42a9-bc7e-31816e80bd72","Type":"ContainerDied","Data":"0ab7fbb0df9faa71bce4f943a9c70a24fd146f7710b82f190ba285737a00416b"}
Jul 18 01:13:08.724607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:08.724110    2554 scope.go:117] "RemoveContainer" containerID="268c37161d739d4abece94dc39ddaba2989d82a083cbb271fa36de5cb13d40ab"
Jul 18 01:13:08.724607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:08.724333    2554 scope.go:117] "RemoveContainer" containerID="0ab7fbb0df9faa71bce4f943a9c70a24fd146f7710b82f190ba285737a00416b"
Jul 18 01:13:08.724607 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:08.724583    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx_odh-ai-gateway-infra(efd4aaec-c28f-42a9-bc7e-31816e80bd72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72"
Jul 18 01:13:09.731101 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:09.731071    2554 scope.go:117] "RemoveContainer" containerID="0ab7fbb0df9faa71bce4f943a9c70a24fd146f7710b82f190ba285737a00416b"
Jul 18 01:13:09.731534 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:09.731273    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx_odh-ai-gateway-infra(efd4aaec-c28f-42a9-bc7e-31816e80bd72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72"
Jul 18 01:13:14.467817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.467783    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"]
Jul 18 01:13:14.468291 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.468235    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd329242-82f2-4844-ba11-7c21d89df8fc" containerName="revoke-keys"
Jul 18 01:13:14.472438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.472421    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"
Jul 18 01:13:14.477065 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.477041    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"]
Jul 18 01:13:14.573393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.573357    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5726713d-4804-4284-b518-51ce4366a89c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-fpz6h\" (UID: \"5726713d-4804-4284-b518-51ce4366a89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"
Jul 18 01:13:14.673876 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.673845    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5726713d-4804-4284-b518-51ce4366a89c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-fpz6h\" (UID: \"5726713d-4804-4284-b518-51ce4366a89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"
Jul 18 01:13:14.674434 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.674414    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5726713d-4804-4284-b518-51ce4366a89c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-fpz6h\" (UID: \"5726713d-4804-4284-b518-51ce4366a89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"
Jul 18 01:13:14.785771 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.785744    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"
Jul 18 01:13:14.914982 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:14.914957    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"]
Jul 18 01:13:14.916842 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:13:14.916814    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5726713d_4804_4284_b518_51ce4366a89c.slice/crio-d2dbbf3f66453549148d4c4c94511eaa45bfcfc8c8d502413fa5b55deb0dd077 WatchSource:0}: Error finding container d2dbbf3f66453549148d4c4c94511eaa45bfcfc8c8d502413fa5b55deb0dd077: Status 404 returned error can't find the container with id d2dbbf3f66453549148d4c4c94511eaa45bfcfc8c8d502413fa5b55deb0dd077
Jul 18 01:13:15.762429 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:15.762393    2554 generic.go:356] "Generic (PLEG): container finished" podID="5726713d-4804-4284-b518-51ce4366a89c" containerID="b9c194a3a6695d578c9add7b14a36e898ff4f9e8754663d7796b6e0318258bde" exitCode=7
Jul 18 01:13:15.762832 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:15.762434    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" event={"ID":"5726713d-4804-4284-b518-51ce4366a89c","Type":"ContainerDied","Data":"b9c194a3a6695d578c9add7b14a36e898ff4f9e8754663d7796b6e0318258bde"}
Jul 18 01:13:15.762832 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:15.762476    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" event={"ID":"5726713d-4804-4284-b518-51ce4366a89c","Type":"ContainerStarted","Data":"d2dbbf3f66453549148d4c4c94511eaa45bfcfc8c8d502413fa5b55deb0dd077"}
Jul 18 01:13:15.762832 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:15.762677    2554 scope.go:117] "RemoveContainer" containerID="b9c194a3a6695d578c9add7b14a36e898ff4f9e8754663d7796b6e0318258bde"
Jul 18 01:13:16.769759 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:16.769723    2554 generic.go:356] "Generic (PLEG): container finished" podID="5726713d-4804-4284-b518-51ce4366a89c" containerID="81fd96a08b25b57b17403d2bcf037ca50e057af13c64884c0207253e7d181d1e" exitCode=7
Jul 18 01:13:16.770201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:16.769804    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" event={"ID":"5726713d-4804-4284-b518-51ce4366a89c","Type":"ContainerDied","Data":"81fd96a08b25b57b17403d2bcf037ca50e057af13c64884c0207253e7d181d1e"}
Jul 18 01:13:16.770201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:16.769856    2554 scope.go:117] "RemoveContainer" containerID="b9c194a3a6695d578c9add7b14a36e898ff4f9e8754663d7796b6e0318258bde"
Jul 18 01:13:16.770201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:16.770007    2554 scope.go:117] "RemoveContainer" containerID="81fd96a08b25b57b17403d2bcf037ca50e057af13c64884c0207253e7d181d1e"
Jul 18 01:13:16.770308 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:16.770243    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-fpz6h_odh-ai-gateway-infra(5726713d-4804-4284-b518-51ce4366a89c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" podUID="5726713d-4804-4284-b518-51ce4366a89c"
Jul 18 01:13:17.776209 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:17.776179    2554 scope.go:117] "RemoveContainer" containerID="81fd96a08b25b57b17403d2bcf037ca50e057af13c64884c0207253e7d181d1e"
Jul 18 01:13:17.776647 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:17.776408    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-fpz6h_odh-ai-gateway-infra(5726713d-4804-4284-b518-51ce4366a89c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" podUID="5726713d-4804-4284-b518-51ce4366a89c"
Jul 18 01:13:18.448457 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.448421    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"]
Jul 18 01:13:18.451713 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.451690    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"
Jul 18 01:13:18.458319 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.458295    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"]
Jul 18 01:13:18.511340 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.511308    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2a948-c524-4daa-83cc-839875648325-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-6npzw\" (UID: \"aed2a948-c524-4daa-83cc-839875648325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"
Jul 18 01:13:18.612507 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.612471    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2a948-c524-4daa-83cc-839875648325-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-6npzw\" (UID: \"aed2a948-c524-4daa-83cc-839875648325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"
Jul 18 01:13:18.613115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.613094    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2a948-c524-4daa-83cc-839875648325-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-6npzw\" (UID: \"aed2a948-c524-4daa-83cc-839875648325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"
Jul 18 01:13:18.763292 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.763251    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"
Jul 18 01:13:18.894112 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:18.894074    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"]
Jul 18 01:13:18.895177 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:13:18.895152    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed2a948_c524_4daa_83cc_839875648325.slice/crio-42984734e086f5c9e77edeb4b1ded365fe5ae9cff94b7ddecff4cd548ae029aa WatchSource:0}: Error finding container 42984734e086f5c9e77edeb4b1ded365fe5ae9cff94b7ddecff4cd548ae029aa: Status 404 returned error can't find the container with id 42984734e086f5c9e77edeb4b1ded365fe5ae9cff94b7ddecff4cd548ae029aa
Jul 18 01:13:19.788390 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:19.788355    2554 generic.go:356] "Generic (PLEG): container finished" podID="aed2a948-c524-4daa-83cc-839875648325" containerID="f0cdc3d1925e75811f755dcfc7cff03c40184bee94a6499dbd4eb4ee74544d8e" exitCode=6
Jul 18 01:13:19.788560 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:19.788442    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" event={"ID":"aed2a948-c524-4daa-83cc-839875648325","Type":"ContainerDied","Data":"f0cdc3d1925e75811f755dcfc7cff03c40184bee94a6499dbd4eb4ee74544d8e"}
Jul 18 01:13:19.788560 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:19.788493    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" event={"ID":"aed2a948-c524-4daa-83cc-839875648325","Type":"ContainerStarted","Data":"42984734e086f5c9e77edeb4b1ded365fe5ae9cff94b7ddecff4cd548ae029aa"}
Jul 18 01:13:19.788658 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:19.788599    2554 scope.go:117] "RemoveContainer" containerID="f0cdc3d1925e75811f755dcfc7cff03c40184bee94a6499dbd4eb4ee74544d8e"
Jul 18 01:13:20.795089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:20.795058    2554 generic.go:356] "Generic (PLEG): container finished" podID="aed2a948-c524-4daa-83cc-839875648325" containerID="e0876ac47e5c4a0a6456e089049e9d7db86867e1e38139fa0028850d1b8695fb" exitCode=6
Jul 18 01:13:20.795574 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:20.795137    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" event={"ID":"aed2a948-c524-4daa-83cc-839875648325","Type":"ContainerDied","Data":"e0876ac47e5c4a0a6456e089049e9d7db86867e1e38139fa0028850d1b8695fb"}
Jul 18 01:13:20.795574 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:20.795195    2554 scope.go:117] "RemoveContainer" containerID="f0cdc3d1925e75811f755dcfc7cff03c40184bee94a6499dbd4eb4ee74544d8e"
Jul 18 01:13:20.795574 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:20.795374    2554 scope.go:117] "RemoveContainer" containerID="e0876ac47e5c4a0a6456e089049e9d7db86867e1e38139fa0028850d1b8695fb"
Jul 18 01:13:20.795744 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:20.795682    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-6npzw_odh-ai-gateway-infra(aed2a948-c524-4daa-83cc-839875648325)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" podUID="aed2a948-c524-4daa-83cc-839875648325"
Jul 18 01:13:21.802646 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:21.802608    2554 scope.go:117] "RemoveContainer" containerID="e0876ac47e5c4a0a6456e089049e9d7db86867e1e38139fa0028850d1b8695fb"
Jul 18 01:13:21.803036 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:21.802829    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-6npzw_odh-ai-gateway-infra(aed2a948-c524-4daa-83cc-839875648325)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" podUID="aed2a948-c524-4daa-83cc-839875648325"
Jul 18 01:13:24.337485 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:24.337448    2554 scope.go:117] "RemoveContainer" containerID="0ab7fbb0df9faa71bce4f943a9c70a24fd146f7710b82f190ba285737a00416b"
Jul 18 01:13:24.818462 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:24.818426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" event={"ID":"efd4aaec-c28f-42a9-bc7e-31816e80bd72","Type":"ContainerStarted","Data":"1a7d608f1b44f4d1a2b19ff29a6ce21d981e6260b3f35ab2443ee8955a3207f6"}
Jul 18 01:13:24.836248 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:24.836200    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" podStartSLOduration=18.836185923 podStartE2EDuration="18.836185923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:13:24.833452496 +0000 UTC m=+1960.959387476" watchObservedRunningTime="2026-07-18 01:13:24.836185923 +0000 UTC m=+1960.962120901"
Jul 18 01:13:25.825579 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:25.825545    2554 generic.go:356] "Generic (PLEG): container finished" podID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerID="1a7d608f1b44f4d1a2b19ff29a6ce21d981e6260b3f35ab2443ee8955a3207f6" exitCode=6
Jul 18 01:13:25.826033 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:25.825637    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" event={"ID":"efd4aaec-c28f-42a9-bc7e-31816e80bd72","Type":"ContainerDied","Data":"1a7d608f1b44f4d1a2b19ff29a6ce21d981e6260b3f35ab2443ee8955a3207f6"}
Jul 18 01:13:25.826033 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:25.825702    2554 scope.go:117] "RemoveContainer" containerID="0ab7fbb0df9faa71bce4f943a9c70a24fd146f7710b82f190ba285737a00416b"
Jul 18 01:13:25.826033 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:25.825990    2554 scope.go:117] "RemoveContainer" containerID="1a7d608f1b44f4d1a2b19ff29a6ce21d981e6260b3f35ab2443ee8955a3207f6"
Jul 18 01:13:25.826265 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:25.826239    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx_odh-ai-gateway-infra(efd4aaec-c28f-42a9-bc7e-31816e80bd72)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72"
Jul 18 01:13:25.842777 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:25.842756    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"]
Jul 18 01:13:26.970273 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:26.970247    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"
Jul 18 01:13:27.096807 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.096726    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4aaec-c28f-42a9-bc7e-31816e80bd72-maas-api-service-ca\") pod \"efd4aaec-c28f-42a9-bc7e-31816e80bd72\" (UID: \"efd4aaec-c28f-42a9-bc7e-31816e80bd72\") "
Jul 18 01:13:27.097100 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.097079    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd4aaec-c28f-42a9-bc7e-31816e80bd72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "efd4aaec-c28f-42a9-bc7e-31816e80bd72" (UID: "efd4aaec-c28f-42a9-bc7e-31816e80bd72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:13:27.197913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.197888    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4aaec-c28f-42a9-bc7e-31816e80bd72-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:13:27.838974 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.838943    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"
Jul 18 01:13:27.838974 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.838966    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx" event={"ID":"efd4aaec-c28f-42a9-bc7e-31816e80bd72","Type":"ContainerDied","Data":"705cb3a17f5bcfa6d15846886e907867d1cd626044240f220f9264aed3e75365"}
Jul 18 01:13:27.839208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.839021    2554 scope.go:117] "RemoveContainer" containerID="1a7d608f1b44f4d1a2b19ff29a6ce21d981e6260b3f35ab2443ee8955a3207f6"
Jul 18 01:13:27.861170 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.861143    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"]
Jul 18 01:13:27.866967 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:27.866946    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-zq5qx"]
Jul 18 01:13:28.336593 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:28.336561    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" path="/var/lib/kubelet/pods/efd4aaec-c28f-42a9-bc7e-31816e80bd72/volumes"
Jul 18 01:13:33.331168 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:33.331136    2554 scope.go:117] "RemoveContainer" containerID="81fd96a08b25b57b17403d2bcf037ca50e057af13c64884c0207253e7d181d1e"
Jul 18 01:13:33.331689 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:33.331406    2554 scope.go:117] "RemoveContainer" containerID="e0876ac47e5c4a0a6456e089049e9d7db86867e1e38139fa0028850d1b8695fb"
Jul 18 01:13:33.871064 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:33.870971    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" event={"ID":"aed2a948-c524-4daa-83cc-839875648325","Type":"ContainerStarted","Data":"fb152c66990f6297a7a3974bb0f481465f0fe18132bb00098f81ffe7c7d3afdd"}
Jul 18 01:13:33.872855 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:33.872831    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" event={"ID":"5726713d-4804-4284-b518-51ce4366a89c","Type":"ContainerStarted","Data":"f8f7084c508c21c8f51f35dc3964364fdb2730bc51c1731ab894f3862cef66bb"}
Jul 18 01:13:33.890360 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:33.890310    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" podStartSLOduration=15.890296205 podStartE2EDuration="15.890296205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:13:33.888289587 +0000 UTC m=+1970.014224569" watchObservedRunningTime="2026-07-18 01:13:33.890296205 +0000 UTC m=+1970.016231184"
Jul 18 01:13:33.906532 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:33.906478    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" podStartSLOduration=19.906460119 podStartE2EDuration="19.906460119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:13:33.904696225 +0000 UTC m=+1970.030631210" watchObservedRunningTime="2026-07-18 01:13:33.906460119 +0000 UTC m=+1970.032395099"
Jul 18 01:13:34.882057 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.882016    2554 generic.go:356] "Generic (PLEG): container finished" podID="aed2a948-c524-4daa-83cc-839875648325" containerID="fb152c66990f6297a7a3974bb0f481465f0fe18132bb00098f81ffe7c7d3afdd" exitCode=6
Jul 18 01:13:34.882490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.882091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" event={"ID":"aed2a948-c524-4daa-83cc-839875648325","Type":"ContainerDied","Data":"fb152c66990f6297a7a3974bb0f481465f0fe18132bb00098f81ffe7c7d3afdd"}
Jul 18 01:13:34.882490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.882151    2554 scope.go:117] "RemoveContainer" containerID="e0876ac47e5c4a0a6456e089049e9d7db86867e1e38139fa0028850d1b8695fb"
Jul 18 01:13:34.882490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.882407    2554 scope.go:117] "RemoveContainer" containerID="fb152c66990f6297a7a3974bb0f481465f0fe18132bb00098f81ffe7c7d3afdd"
Jul 18 01:13:34.882711 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:34.882686    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-6npzw_odh-ai-gateway-infra(aed2a948-c524-4daa-83cc-839875648325)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" podUID="aed2a948-c524-4daa-83cc-839875648325"
Jul 18 01:13:34.884090 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.884070    2554 generic.go:356] "Generic (PLEG): container finished" podID="5726713d-4804-4284-b518-51ce4366a89c" containerID="f8f7084c508c21c8f51f35dc3964364fdb2730bc51c1731ab894f3862cef66bb" exitCode=7
Jul 18 01:13:34.884205 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.884137    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" event={"ID":"5726713d-4804-4284-b518-51ce4366a89c","Type":"ContainerDied","Data":"f8f7084c508c21c8f51f35dc3964364fdb2730bc51c1731ab894f3862cef66bb"}
Jul 18 01:13:34.884384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.884371    2554 scope.go:117] "RemoveContainer" containerID="f8f7084c508c21c8f51f35dc3964364fdb2730bc51c1731ab894f3862cef66bb"
Jul 18 01:13:34.884559 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:13:34.884543    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-fpz6h_odh-ai-gateway-infra(5726713d-4804-4284-b518-51ce4366a89c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" podUID="5726713d-4804-4284-b518-51ce4366a89c"
Jul 18 01:13:34.893217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.893200    2554 scope.go:117] "RemoveContainer" containerID="81fd96a08b25b57b17403d2bcf037ca50e057af13c64884c0207253e7d181d1e"
Jul 18 01:13:34.904311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.904290    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"]
Jul 18 01:13:34.925841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:34.925813    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"]
Jul 18 01:13:36.048010 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.047985    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"
Jul 18 01:13:36.049951 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.049934    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"
Jul 18 01:13:36.186477 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.186369    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2a948-c524-4daa-83cc-839875648325-maas-api-service-ca\") pod \"aed2a948-c524-4daa-83cc-839875648325\" (UID: \"aed2a948-c524-4daa-83cc-839875648325\") "
Jul 18 01:13:36.186477 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.186457    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5726713d-4804-4284-b518-51ce4366a89c-maas-api-service-ca\") pod \"5726713d-4804-4284-b518-51ce4366a89c\" (UID: \"5726713d-4804-4284-b518-51ce4366a89c\") "
Jul 18 01:13:36.186827 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.186801    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed2a948-c524-4daa-83cc-839875648325-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aed2a948-c524-4daa-83cc-839875648325" (UID: "aed2a948-c524-4daa-83cc-839875648325"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:13:36.186871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.186834    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5726713d-4804-4284-b518-51ce4366a89c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5726713d-4804-4284-b518-51ce4366a89c" (UID: "5726713d-4804-4284-b518-51ce4366a89c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:13:36.287599 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.287561    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2a948-c524-4daa-83cc-839875648325-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:13:36.287599 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.287593    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5726713d-4804-4284-b518-51ce4366a89c-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:13:36.898488 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.898461    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"
Jul 18 01:13:36.898684 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.898488    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h" event={"ID":"5726713d-4804-4284-b518-51ce4366a89c","Type":"ContainerDied","Data":"d2dbbf3f66453549148d4c4c94511eaa45bfcfc8c8d502413fa5b55deb0dd077"}
Jul 18 01:13:36.898684 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.898579    2554 scope.go:117] "RemoveContainer" containerID="f8f7084c508c21c8f51f35dc3964364fdb2730bc51c1731ab894f3862cef66bb"
Jul 18 01:13:36.899928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.899799    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw" event={"ID":"aed2a948-c524-4daa-83cc-839875648325","Type":"ContainerDied","Data":"42984734e086f5c9e77edeb4b1ded365fe5ae9cff94b7ddecff4cd548ae029aa"}
Jul 18 01:13:36.899928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.899849    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"
Jul 18 01:13:36.906980 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.906963    2554 scope.go:117] "RemoveContainer" containerID="fb152c66990f6297a7a3974bb0f481465f0fe18132bb00098f81ffe7c7d3afdd"
Jul 18 01:13:36.916084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.916066    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"]
Jul 18 01:13:36.919673 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.919651    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fpz6h"]
Jul 18 01:13:36.929202 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.929178    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"]
Jul 18 01:13:36.934682 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:36.934663    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-6npzw"]
Jul 18 01:13:38.336168 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:38.336131    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5726713d-4804-4284-b518-51ce4366a89c" path="/var/lib/kubelet/pods/5726713d-4804-4284-b518-51ce4366a89c/volumes"
Jul 18 01:13:38.336546 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:13:38.336478    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed2a948-c524-4daa-83cc-839875648325" path="/var/lib/kubelet/pods/aed2a948-c524-4daa-83cc-839875648325/volumes"
Jul 18 01:14:06.507427 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507390    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"]
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507793    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507806    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507815    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507820    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507829    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507835    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507840    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507845    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507851    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507856    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507865    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:06.507885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507870    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507893    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507898    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507946    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507953    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507960    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507967    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507973    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507978    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:06.508412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.507985    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:06.511133 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.511114    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"
Jul 18 01:14:06.513455 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.513430    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:14:06.518534 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.518506    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"]
Jul 18 01:14:06.580731 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.580700    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd1a4d-7c46-4b1d-b0fa-e334ced21955-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j\" (UID: \"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"
Jul 18 01:14:06.681778 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.681738    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd1a4d-7c46-4b1d-b0fa-e334ced21955-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j\" (UID: \"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"
Jul 18 01:14:06.682348 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.682327    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd1a4d-7c46-4b1d-b0fa-e334ced21955-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j\" (UID: \"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"
Jul 18 01:14:06.823958 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.823862    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"
Jul 18 01:14:06.956394 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:06.956354    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"]
Jul 18 01:14:06.957412 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:14:06.957385    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdd1a4d_7c46_4b1d_b0fa_e334ced21955.slice/crio-fcb83594d574e3ab398a1ec58b82ac2973fbdf2c0616b4867f14151e231eca8c WatchSource:0}: Error finding container fcb83594d574e3ab398a1ec58b82ac2973fbdf2c0616b4867f14151e231eca8c: Status 404 returned error can't find the container with id fcb83594d574e3ab398a1ec58b82ac2973fbdf2c0616b4867f14151e231eca8c
Jul 18 01:14:07.058026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:07.057994    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" event={"ID":"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955","Type":"ContainerStarted","Data":"fcb83594d574e3ab398a1ec58b82ac2973fbdf2c0616b4867f14151e231eca8c"}
Jul 18 01:14:08.064497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.064463    2554 generic.go:356] "Generic (PLEG): container finished" podID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerID="462919efa196472bed780a00fd14ea9fe180a15dc727a3b8107858f2b9477f6f" exitCode=6
Jul 18 01:14:08.064916 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.064547    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" event={"ID":"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955","Type":"ContainerDied","Data":"462919efa196472bed780a00fd14ea9fe180a15dc727a3b8107858f2b9477f6f"}
Jul 18 01:14:08.064916 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.064741    2554 scope.go:117] "RemoveContainer" containerID="462919efa196472bed780a00fd14ea9fe180a15dc727a3b8107858f2b9477f6f"
Jul 18 01:14:08.482432 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.482399    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"]
Jul 18 01:14:08.482947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.482926    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:08.482947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.482947    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd4aaec-c28f-42a9-bc7e-31816e80bd72" containerName="revoke-keys"
Jul 18 01:14:08.483122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.482963    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:08.483122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.482969    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:08.483122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.483046    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed2a948-c524-4daa-83cc-839875648325" containerName="revoke-keys"
Jul 18 01:14:08.483122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.483058    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5726713d-4804-4284-b518-51ce4366a89c" containerName="revoke-keys"
Jul 18 01:14:08.486090 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.486073    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"
Jul 18 01:14:08.492355 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.492312    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"]
Jul 18 01:14:08.600968 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.600880    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47dd26cc-14ff-449a-a1df-db8d6d860359-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-75sks\" (UID: \"47dd26cc-14ff-449a-a1df-db8d6d860359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"
Jul 18 01:14:08.702345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.702318    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47dd26cc-14ff-449a-a1df-db8d6d860359-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-75sks\" (UID: \"47dd26cc-14ff-449a-a1df-db8d6d860359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"
Jul 18 01:14:08.703016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.702989    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47dd26cc-14ff-449a-a1df-db8d6d860359-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-75sks\" (UID: \"47dd26cc-14ff-449a-a1df-db8d6d860359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"
Jul 18 01:14:08.798569 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.798529    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"
Jul 18 01:14:08.927316 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:08.927290    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"]
Jul 18 01:14:08.929486 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:14:08.929462    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dd26cc_14ff_449a_a1df_db8d6d860359.slice/crio-abc37aa969ccc487608919415bfaab5a0f599538a6953aea4c3f342caf53e348 WatchSource:0}: Error finding container abc37aa969ccc487608919415bfaab5a0f599538a6953aea4c3f342caf53e348: Status 404 returned error can't find the container with id abc37aa969ccc487608919415bfaab5a0f599538a6953aea4c3f342caf53e348
Jul 18 01:14:09.071139 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:09.071104    2554 generic.go:356] "Generic (PLEG): container finished" podID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerID="acb626525c29ff4cce77bc243ccd4a7e4d741d6cca468b98b85ad010ca82c37b" exitCode=6
Jul 18 01:14:09.071575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:09.071178    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" event={"ID":"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955","Type":"ContainerDied","Data":"acb626525c29ff4cce77bc243ccd4a7e4d741d6cca468b98b85ad010ca82c37b"}
Jul 18 01:14:09.071575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:09.071254    2554 scope.go:117] "RemoveContainer" containerID="462919efa196472bed780a00fd14ea9fe180a15dc727a3b8107858f2b9477f6f"
Jul 18 01:14:09.071575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:09.071487    2554 scope.go:117] "RemoveContainer" containerID="acb626525c29ff4cce77bc243ccd4a7e4d741d6cca468b98b85ad010ca82c37b"
Jul 18 01:14:09.071796 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:09.071776    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j_odh-ai-gateway-infra(3bdd1a4d-7c46-4b1d-b0fa-e334ced21955)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955"
Jul 18 01:14:09.072565 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:09.072366    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" event={"ID":"47dd26cc-14ff-449a-a1df-db8d6d860359","Type":"ContainerStarted","Data":"abc37aa969ccc487608919415bfaab5a0f599538a6953aea4c3f342caf53e348"}
Jul 18 01:14:10.079023 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:10.078983    2554 scope.go:117] "RemoveContainer" containerID="acb626525c29ff4cce77bc243ccd4a7e4d741d6cca468b98b85ad010ca82c37b"
Jul 18 01:14:10.079433 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:10.079234    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j_odh-ai-gateway-infra(3bdd1a4d-7c46-4b1d-b0fa-e334ced21955)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955"
Jul 18 01:14:10.080180 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:10.080157    2554 generic.go:356] "Generic (PLEG): container finished" podID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerID="632de679151417ba0a3fbae4db87a08169dc9c4f7690bcffae4436fec74af6e9" exitCode=6
Jul 18 01:14:10.080279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:10.080189    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" event={"ID":"47dd26cc-14ff-449a-a1df-db8d6d860359","Type":"ContainerDied","Data":"632de679151417ba0a3fbae4db87a08169dc9c4f7690bcffae4436fec74af6e9"}
Jul 18 01:14:10.080352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:10.080340    2554 scope.go:117] "RemoveContainer" containerID="632de679151417ba0a3fbae4db87a08169dc9c4f7690bcffae4436fec74af6e9"
Jul 18 01:14:11.093184 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:11.093147    2554 generic.go:356] "Generic (PLEG): container finished" podID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerID="65fd7a400b46ac06232ee220ea8c0b088570fe10556ed987cde36fe8e7da1415" exitCode=6
Jul 18 01:14:11.093682 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:11.093227    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" event={"ID":"47dd26cc-14ff-449a-a1df-db8d6d860359","Type":"ContainerDied","Data":"65fd7a400b46ac06232ee220ea8c0b088570fe10556ed987cde36fe8e7da1415"}
Jul 18 01:14:11.093682 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:11.093301    2554 scope.go:117] "RemoveContainer" containerID="632de679151417ba0a3fbae4db87a08169dc9c4f7690bcffae4436fec74af6e9"
Jul 18 01:14:11.093682 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:11.093469    2554 scope.go:117] "RemoveContainer" containerID="65fd7a400b46ac06232ee220ea8c0b088570fe10556ed987cde36fe8e7da1415"
Jul 18 01:14:11.093862 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:11.093756    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-75sks_odh-ai-gateway-infra(47dd26cc-14ff-449a-a1df-db8d6d860359)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359"
Jul 18 01:14:12.099629 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:12.099583    2554 scope.go:117] "RemoveContainer" containerID="65fd7a400b46ac06232ee220ea8c0b088570fe10556ed987cde36fe8e7da1415"
Jul 18 01:14:12.100013 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:12.099813    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-75sks_odh-ai-gateway-infra(47dd26cc-14ff-449a-a1df-db8d6d860359)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359"
Jul 18 01:14:14.512251 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.512219    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"]
Jul 18 01:14:14.517016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.516992    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"
Jul 18 01:14:14.521792 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.521758    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"]
Jul 18 01:14:14.654854 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.654816    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387e6c6-3751-4da1-bab0-cbed3b6c2153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-spmn9\" (UID: \"a387e6c6-3751-4da1-bab0-cbed3b6c2153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"
Jul 18 01:14:14.755962 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.755929    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387e6c6-3751-4da1-bab0-cbed3b6c2153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-spmn9\" (UID: \"a387e6c6-3751-4da1-bab0-cbed3b6c2153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"
Jul 18 01:14:14.756519 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.756501    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387e6c6-3751-4da1-bab0-cbed3b6c2153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-spmn9\" (UID: \"a387e6c6-3751-4da1-bab0-cbed3b6c2153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"
Jul 18 01:14:14.829763 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.829698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"
Jul 18 01:14:14.959647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:14.959548    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"]
Jul 18 01:14:14.961973 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:14:14.961940    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda387e6c6_3751_4da1_bab0_cbed3b6c2153.slice/crio-823243e3a1a8c6837396dd43f88e517e3cd26654ecb935fd72e88d81876cfbf8 WatchSource:0}: Error finding container 823243e3a1a8c6837396dd43f88e517e3cd26654ecb935fd72e88d81876cfbf8: Status 404 returned error can't find the container with id 823243e3a1a8c6837396dd43f88e517e3cd26654ecb935fd72e88d81876cfbf8
Jul 18 01:14:15.113807 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:15.113737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" event={"ID":"a387e6c6-3751-4da1-bab0-cbed3b6c2153","Type":"ContainerStarted","Data":"823243e3a1a8c6837396dd43f88e517e3cd26654ecb935fd72e88d81876cfbf8"}
Jul 18 01:14:16.125885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:16.125844    2554 generic.go:356] "Generic (PLEG): container finished" podID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerID="291d69512da163d112a34d7d12170b26dc095e2a70b5aaa4f1a2d1aa38e7f279" exitCode=7
Jul 18 01:14:16.126287 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:16.125919    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" event={"ID":"a387e6c6-3751-4da1-bab0-cbed3b6c2153","Type":"ContainerDied","Data":"291d69512da163d112a34d7d12170b26dc095e2a70b5aaa4f1a2d1aa38e7f279"}
Jul 18 01:14:16.126287 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:16.126180    2554 scope.go:117] "RemoveContainer" containerID="291d69512da163d112a34d7d12170b26dc095e2a70b5aaa4f1a2d1aa38e7f279"
Jul 18 01:14:17.132144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:17.132107    2554 generic.go:356] "Generic (PLEG): container finished" podID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerID="57f435f51115c827cc8150041bf70d5a9d94c71cbaf04b329de4e1cf3e1abc4a" exitCode=7
Jul 18 01:14:17.132641 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:17.132185    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" event={"ID":"a387e6c6-3751-4da1-bab0-cbed3b6c2153","Type":"ContainerDied","Data":"57f435f51115c827cc8150041bf70d5a9d94c71cbaf04b329de4e1cf3e1abc4a"}
Jul 18 01:14:17.132641 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:17.132246    2554 scope.go:117] "RemoveContainer" containerID="291d69512da163d112a34d7d12170b26dc095e2a70b5aaa4f1a2d1aa38e7f279"
Jul 18 01:14:17.132641 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:17.132445    2554 scope.go:117] "RemoveContainer" containerID="57f435f51115c827cc8150041bf70d5a9d94c71cbaf04b329de4e1cf3e1abc4a"
Jul 18 01:14:17.132804 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:17.132674    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-spmn9_odh-ai-gateway-infra(a387e6c6-3751-4da1-bab0-cbed3b6c2153)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153"
Jul 18 01:14:18.138780 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:18.138756    2554 scope.go:117] "RemoveContainer" containerID="57f435f51115c827cc8150041bf70d5a9d94c71cbaf04b329de4e1cf3e1abc4a"
Jul 18 01:14:18.139152 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:18.138942    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-spmn9_odh-ai-gateway-infra(a387e6c6-3751-4da1-bab0-cbed3b6c2153)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153"
Jul 18 01:14:24.333207 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:24.333175    2554 scope.go:117] "RemoveContainer" containerID="acb626525c29ff4cce77bc243ccd4a7e4d741d6cca468b98b85ad010ca82c37b"
Jul 18 01:14:25.175247 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:25.175168    2554 generic.go:356] "Generic (PLEG): container finished" podID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerID="3976224ef0e6569ab474e033ef8f12ea41681e7de104b369dc8a5a19bfcad2d5" exitCode=6
Jul 18 01:14:25.175247 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:25.175227    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" event={"ID":"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955","Type":"ContainerDied","Data":"3976224ef0e6569ab474e033ef8f12ea41681e7de104b369dc8a5a19bfcad2d5"}
Jul 18 01:14:25.175435 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:25.175278    2554 scope.go:117] "RemoveContainer" containerID="acb626525c29ff4cce77bc243ccd4a7e4d741d6cca468b98b85ad010ca82c37b"
Jul 18 01:14:25.175580 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:25.175564    2554 scope.go:117] "RemoveContainer" containerID="3976224ef0e6569ab474e033ef8f12ea41681e7de104b369dc8a5a19bfcad2d5"
Jul 18 01:14:25.175858 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:25.175837    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j_odh-ai-gateway-infra(3bdd1a4d-7c46-4b1d-b0fa-e334ced21955)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955"
Jul 18 01:14:25.330969 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:25.330942    2554 scope.go:117] "RemoveContainer" containerID="65fd7a400b46ac06232ee220ea8c0b088570fe10556ed987cde36fe8e7da1415"
Jul 18 01:14:26.183019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.182934    2554 generic.go:356] "Generic (PLEG): container finished" podID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerID="0483ba0e305c78271204670e0bc9074cb43eb568d4c2fb5b0f7f22eb76b933a7" exitCode=6
Jul 18 01:14:26.183019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.182985    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" event={"ID":"47dd26cc-14ff-449a-a1df-db8d6d860359","Type":"ContainerDied","Data":"0483ba0e305c78271204670e0bc9074cb43eb568d4c2fb5b0f7f22eb76b933a7"}
Jul 18 01:14:26.183547 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.183030    2554 scope.go:117] "RemoveContainer" containerID="65fd7a400b46ac06232ee220ea8c0b088570fe10556ed987cde36fe8e7da1415"
Jul 18 01:14:26.183547 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.183340    2554 scope.go:117] "RemoveContainer" containerID="0483ba0e305c78271204670e0bc9074cb43eb568d4c2fb5b0f7f22eb76b933a7"
Jul 18 01:14:26.183690 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:26.183666    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-75sks_odh-ai-gateway-infra(47dd26cc-14ff-449a-a1df-db8d6d860359)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359"
Jul 18 01:14:26.205739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.205707    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"]
Jul 18 01:14:26.343261 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.343237    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"
Jul 18 01:14:26.473072 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.472991    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd1a4d-7c46-4b1d-b0fa-e334ced21955-maas-api-service-ca\") pod \"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955\" (UID: \"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955\") "
Jul 18 01:14:26.473385 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.473354    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdd1a4d-7c46-4b1d-b0fa-e334ced21955-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" (UID: "3bdd1a4d-7c46-4b1d-b0fa-e334ced21955"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:14:26.573871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:26.573843    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bdd1a4d-7c46-4b1d-b0fa-e334ced21955-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:14:27.189124 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.189096    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"
Jul 18 01:14:27.189124 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.189105    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j" event={"ID":"3bdd1a4d-7c46-4b1d-b0fa-e334ced21955","Type":"ContainerDied","Data":"fcb83594d574e3ab398a1ec58b82ac2973fbdf2c0616b4867f14151e231eca8c"}
Jul 18 01:14:27.189676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.189166    2554 scope.go:117] "RemoveContainer" containerID="3976224ef0e6569ab474e033ef8f12ea41681e7de104b369dc8a5a19bfcad2d5"
Jul 18 01:14:27.211682 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.211656    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"]
Jul 18 01:14:27.218513 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.218488    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-7pp8j"]
Jul 18 01:14:27.220175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.220151    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"]
Jul 18 01:14:27.359588 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.359561    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"
Jul 18 01:14:27.381247 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.381209    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47dd26cc-14ff-449a-a1df-db8d6d860359-maas-api-service-ca\") pod \"47dd26cc-14ff-449a-a1df-db8d6d860359\" (UID: \"47dd26cc-14ff-449a-a1df-db8d6d860359\") "
Jul 18 01:14:27.381557 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.381529    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47dd26cc-14ff-449a-a1df-db8d6d860359-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47dd26cc-14ff-449a-a1df-db8d6d860359" (UID: "47dd26cc-14ff-449a-a1df-db8d6d860359"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:14:27.482668 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:27.482561    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47dd26cc-14ff-449a-a1df-db8d6d860359-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:14:28.197565 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:28.197535    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"
Jul 18 01:14:28.198012 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:28.197535    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks" event={"ID":"47dd26cc-14ff-449a-a1df-db8d6d860359","Type":"ContainerDied","Data":"abc37aa969ccc487608919415bfaab5a0f599538a6953aea4c3f342caf53e348"}
Jul 18 01:14:28.198012 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:28.197676    2554 scope.go:117] "RemoveContainer" containerID="0483ba0e305c78271204670e0bc9074cb43eb568d4c2fb5b0f7f22eb76b933a7"
Jul 18 01:14:28.218252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:28.218228    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"]
Jul 18 01:14:28.222101 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:28.222072    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-75sks"]
Jul 18 01:14:28.336140 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:28.336109    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" path="/var/lib/kubelet/pods/3bdd1a4d-7c46-4b1d-b0fa-e334ced21955/volumes"
Jul 18 01:14:28.336501 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:28.336487    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" path="/var/lib/kubelet/pods/47dd26cc-14ff-449a-a1df-db8d6d860359/volumes"
Jul 18 01:14:30.330638 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:30.330587    2554 scope.go:117] "RemoveContainer" containerID="57f435f51115c827cc8150041bf70d5a9d94c71cbaf04b329de4e1cf3e1abc4a"
Jul 18 01:14:31.215253 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:31.215168    2554 generic.go:356] "Generic (PLEG): container finished" podID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerID="78ed43f772e4fe33e19a4f21fd32524ec349bf2440156de81c833b4e1e73ba51" exitCode=7
Jul 18 01:14:31.215389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:31.215246    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" event={"ID":"a387e6c6-3751-4da1-bab0-cbed3b6c2153","Type":"ContainerDied","Data":"78ed43f772e4fe33e19a4f21fd32524ec349bf2440156de81c833b4e1e73ba51"}
Jul 18 01:14:31.215389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:31.215310    2554 scope.go:117] "RemoveContainer" containerID="57f435f51115c827cc8150041bf70d5a9d94c71cbaf04b329de4e1cf3e1abc4a"
Jul 18 01:14:31.215660 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:31.215608    2554 scope.go:117] "RemoveContainer" containerID="78ed43f772e4fe33e19a4f21fd32524ec349bf2440156de81c833b4e1e73ba51"
Jul 18 01:14:31.215885 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:14:31.215860    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-spmn9_odh-ai-gateway-infra(a387e6c6-3751-4da1-bab0-cbed3b6c2153)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153"
Jul 18 01:14:32.246055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:32.246016    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"]
Jul 18 01:14:32.382846 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:32.382821    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"
Jul 18 01:14:32.420453 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:32.420425    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387e6c6-3751-4da1-bab0-cbed3b6c2153-maas-api-service-ca\") pod \"a387e6c6-3751-4da1-bab0-cbed3b6c2153\" (UID: \"a387e6c6-3751-4da1-bab0-cbed3b6c2153\") "
Jul 18 01:14:32.423698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:32.421247    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a387e6c6-3751-4da1-bab0-cbed3b6c2153-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a387e6c6-3751-4da1-bab0-cbed3b6c2153" (UID: "a387e6c6-3751-4da1-bab0-cbed3b6c2153"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:14:32.522789 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:32.522720    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387e6c6-3751-4da1-bab0-cbed3b6c2153-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:14:33.228036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:33.228007    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"
Jul 18 01:14:33.228036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:33.228018    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9" event={"ID":"a387e6c6-3751-4da1-bab0-cbed3b6c2153","Type":"ContainerDied","Data":"823243e3a1a8c6837396dd43f88e517e3cd26654ecb935fd72e88d81876cfbf8"}
Jul 18 01:14:33.228250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:33.228110    2554 scope.go:117] "RemoveContainer" containerID="78ed43f772e4fe33e19a4f21fd32524ec349bf2440156de81c833b4e1e73ba51"
Jul 18 01:14:33.248212 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:33.248148    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"]
Jul 18 01:14:33.250472 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:33.250454    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-spmn9"]
Jul 18 01:14:34.336019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:34.335986    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" path="/var/lib/kubelet/pods/a387e6c6-3751-4da1-bab0-cbed3b6c2153/volumes"
Jul 18 01:14:58.526307 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526254    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"]
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526662    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526673    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526692    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526698    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526704    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526710    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526716    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526721    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526731    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526736    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526747    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526753    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526760    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526766    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526815    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526824    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526829    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526836    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526841    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526846    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a387e6c6-3751-4da1-bab0-cbed3b6c2153" containerName="revoke-keys"
Jul 18 01:14:58.526868 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.526851    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:14:58.529984 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.529964    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"
Jul 18 01:14:58.532311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.532286    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:14:58.536549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.536526    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"]
Jul 18 01:14:58.674080 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.674044    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8\" (UID: \"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"
Jul 18 01:14:58.774994 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.774959    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8\" (UID: \"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"
Jul 18 01:14:58.775691 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.775673    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8\" (UID: \"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"
Jul 18 01:14:58.842967 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.842884    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"
Jul 18 01:14:58.971280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:58.971249    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"]
Jul 18 01:14:58.973598 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:14:58.973564    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5ce8d7_97fa_4289_bfb0_7f9d428ac4f6.slice/crio-2c62b7a1294655e10bd891b1c029a5a7908b583b3fef63240a2ec245989ef616 WatchSource:0}: Error finding container 2c62b7a1294655e10bd891b1c029a5a7908b583b3fef63240a2ec245989ef616: Status 404 returned error can't find the container with id 2c62b7a1294655e10bd891b1c029a5a7908b583b3fef63240a2ec245989ef616
Jul 18 01:14:59.366398 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:59.366368    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" event={"ID":"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6","Type":"ContainerStarted","Data":"7a8d742c69242b559187b3b5ea71b599876654859969627d591ff0b1c633e16b"}
Jul 18 01:14:59.366569 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:59.366416    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" event={"ID":"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6","Type":"ContainerStarted","Data":"2c62b7a1294655e10bd891b1c029a5a7908b583b3fef63240a2ec245989ef616"}
Jul 18 01:14:59.382986 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:14:59.382938    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" podStartSLOduration=1.382922386 podStartE2EDuration="1.382922386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:14:59.38074138 +0000 UTC m=+2055.506676362" watchObservedRunningTime="2026-07-18 01:14:59.382922386 +0000 UTC m=+2055.508857365"
Jul 18 01:15:00.140525 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.140478    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k"]
Jul 18 01:15:00.140990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.140973    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:15:00.141043 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.140992    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:15:00.141043 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.141017    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:15:00.141043 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.141024    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:15:00.141157 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.141081    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="47dd26cc-14ff-449a-a1df-db8d6d860359" containerName="revoke-keys"
Jul 18 01:15:00.141157 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.141090    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bdd1a4d-7c46-4b1d-b0fa-e334ced21955" containerName="revoke-keys"
Jul 18 01:15:00.144093 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.144076    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"]
Jul 18 01:15:00.144208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.144190    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k"
Jul 18 01:15:00.147274 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.147258    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"
Jul 18 01:15:00.151909 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.151560    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k"]
Jul 18 01:15:00.155373 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.155348    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"]
Jul 18 01:15:00.156900 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.156882    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k"
Jul 18 01:15:00.164961 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.164942    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"
Jul 18 01:15:00.317748 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.317722    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"]
Jul 18 01:15:00.319544 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:15:00.319513    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031ca428_9136_4309_b972_e8529e376ad8.slice/crio-cda0b846901ead171541b5e95279e6a5a93b02a809f71bbd9119665aecac978a WatchSource:0}: Error finding container cda0b846901ead171541b5e95279e6a5a93b02a809f71bbd9119665aecac978a: Status 404 returned error can't find the container with id cda0b846901ead171541b5e95279e6a5a93b02a809f71bbd9119665aecac978a
Jul 18 01:15:00.372701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.372674    2554 generic.go:356] "Generic (PLEG): container finished" podID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerID="7a8d742c69242b559187b3b5ea71b599876654859969627d591ff0b1c633e16b" exitCode=6
Jul 18 01:15:00.372799 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.372755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" event={"ID":"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6","Type":"ContainerDied","Data":"7a8d742c69242b559187b3b5ea71b599876654859969627d591ff0b1c633e16b"}
Jul 18 01:15:00.372981 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.372961    2554 scope.go:117] "RemoveContainer" containerID="7a8d742c69242b559187b3b5ea71b599876654859969627d591ff0b1c633e16b"
Jul 18 01:15:00.374015 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.373988    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" event={"ID":"031ca428-9136-4309-b972-e8529e376ad8","Type":"ContainerStarted","Data":"cda0b846901ead171541b5e95279e6a5a93b02a809f71bbd9119665aecac978a"}
Jul 18 01:15:00.501978 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:00.501956    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k"]
Jul 18 01:15:00.504261 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:15:00.504235    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12b05e5_679d_46b2_be7d_fef5d13239fa.slice/crio-25e90296b25f3a82217fd9760ecbf01f5eacb4ce85ccadfcc3662970a71161a7 WatchSource:0}: Error finding container 25e90296b25f3a82217fd9760ecbf01f5eacb4ce85ccadfcc3662970a71161a7: Status 404 returned error can't find the container with id 25e90296b25f3a82217fd9760ecbf01f5eacb4ce85ccadfcc3662970a71161a7
Jul 18 01:15:01.380866 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.380785    2554 generic.go:356] "Generic (PLEG): container finished" podID="031ca428-9136-4309-b972-e8529e376ad8" containerID="6a2c322506f2d40b6921f20becabf2c00ffc99caf58d71f1d7ea456869d5e45a" exitCode=7
Jul 18 01:15:01.381292 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.380868    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" event={"ID":"031ca428-9136-4309-b972-e8529e376ad8","Type":"ContainerDied","Data":"6a2c322506f2d40b6921f20becabf2c00ffc99caf58d71f1d7ea456869d5e45a"}
Jul 18 01:15:01.381292 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.381018    2554 scope.go:117] "RemoveContainer" containerID="6a2c322506f2d40b6921f20becabf2c00ffc99caf58d71f1d7ea456869d5e45a"
Jul 18 01:15:01.382902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.382879    2554 generic.go:356] "Generic (PLEG): container finished" podID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerID="2463499dd74c3d8aa4f7617472326f49797f0e77d531c031401719a053d7d54f" exitCode=6
Jul 18 01:15:01.383013 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.382953    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" event={"ID":"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6","Type":"ContainerDied","Data":"2463499dd74c3d8aa4f7617472326f49797f0e77d531c031401719a053d7d54f"}
Jul 18 01:15:01.383070 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.383034    2554 scope.go:117] "RemoveContainer" containerID="7a8d742c69242b559187b3b5ea71b599876654859969627d591ff0b1c633e16b"
Jul 18 01:15:01.383275 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.383219    2554 scope.go:117] "RemoveContainer" containerID="2463499dd74c3d8aa4f7617472326f49797f0e77d531c031401719a053d7d54f"
Jul 18 01:15:01.383462 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:01.383436    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8_odh-ai-gateway-infra(5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6"
Jul 18 01:15:01.384678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.384655    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k" event={"ID":"a12b05e5-679d-46b2-be7d-fef5d13239fa","Type":"ContainerStarted","Data":"2fb39a6e16d418be8f3ad4e403c0cd69a895c549956002bdd49bc5885d7b97a4"}
Jul 18 01:15:01.384789 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.384702    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k" event={"ID":"a12b05e5-679d-46b2-be7d-fef5d13239fa","Type":"ContainerStarted","Data":"25e90296b25f3a82217fd9760ecbf01f5eacb4ce85ccadfcc3662970a71161a7"}
Jul 18 01:15:01.424522 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:01.424474    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k" podStartSLOduration=1.424456048 podStartE2EDuration="1.424456048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:15:01.422501711 +0000 UTC m=+2057.548436691" watchObservedRunningTime="2026-07-18 01:15:01.424456048 +0000 UTC m=+2057.550391063"
Jul 18 01:15:02.391512 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:02.391409    2554 scope.go:117] "RemoveContainer" containerID="2463499dd74c3d8aa4f7617472326f49797f0e77d531c031401719a053d7d54f"
Jul 18 01:15:02.392002 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:02.391732    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8_odh-ai-gateway-infra(5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6"
Jul 18 01:15:02.392610 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:02.392589    2554 generic.go:356] "Generic (PLEG): container finished" podID="a12b05e5-679d-46b2-be7d-fef5d13239fa" containerID="2fb39a6e16d418be8f3ad4e403c0cd69a895c549956002bdd49bc5885d7b97a4" exitCode=0
Jul 18 01:15:02.392716 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:02.392647    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k" event={"ID":"a12b05e5-679d-46b2-be7d-fef5d13239fa","Type":"ContainerDied","Data":"2fb39a6e16d418be8f3ad4e403c0cd69a895c549956002bdd49bc5885d7b97a4"}
Jul 18 01:15:02.394351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:02.394325    2554 generic.go:356] "Generic (PLEG): container finished" podID="031ca428-9136-4309-b972-e8529e376ad8" containerID="c8b937b1904aa4795ad8ff588c88f780760042fc5329ae564440d0e1317d14d2" exitCode=7
Jul 18 01:15:02.394471 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:02.394359    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" event={"ID":"031ca428-9136-4309-b972-e8529e376ad8","Type":"ContainerDied","Data":"c8b937b1904aa4795ad8ff588c88f780760042fc5329ae564440d0e1317d14d2"}
Jul 18 01:15:02.394471 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:02.394399    2554 scope.go:117] "RemoveContainer" containerID="6a2c322506f2d40b6921f20becabf2c00ffc99caf58d71f1d7ea456869d5e45a"
Jul 18 01:15:02.394594 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:02.394579    2554 scope.go:117] "RemoveContainer" containerID="c8b937b1904aa4795ad8ff588c88f780760042fc5329ae564440d0e1317d14d2"
Jul 18 01:15:02.394876 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:02.394852    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd_odh-ai-gateway-infra(031ca428-9136-4309-b972-e8529e376ad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" podUID="031ca428-9136-4309-b972-e8529e376ad8"
Jul 18 01:15:03.400701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:03.400674    2554 scope.go:117] "RemoveContainer" containerID="c8b937b1904aa4795ad8ff588c88f780760042fc5329ae564440d0e1317d14d2"
Jul 18 01:15:03.401086 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:03.400938    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd_odh-ai-gateway-infra(031ca428-9136-4309-b972-e8529e376ad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" podUID="031ca428-9136-4309-b972-e8529e376ad8"
Jul 18 01:15:03.554360 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:03.554337    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k"
Jul 18 01:15:04.406837 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.406814    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k"
Jul 18 01:15:04.406837 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.406820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738955-sqz2k" event={"ID":"a12b05e5-679d-46b2-be7d-fef5d13239fa","Type":"ContainerDied","Data":"25e90296b25f3a82217fd9760ecbf01f5eacb4ce85ccadfcc3662970a71161a7"}
Jul 18 01:15:04.407250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.406865    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e90296b25f3a82217fd9760ecbf01f5eacb4ce85ccadfcc3662970a71161a7"
Jul 18 01:15:04.550445 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.550407    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"]
Jul 18 01:15:04.550925 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.550905    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12b05e5-679d-46b2-be7d-fef5d13239fa" containerName="cleanup"
Jul 18 01:15:04.551032 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.550926    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12b05e5-679d-46b2-be7d-fef5d13239fa" containerName="cleanup"
Jul 18 01:15:04.551032 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.551014    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12b05e5-679d-46b2-be7d-fef5d13239fa" containerName="cleanup"
Jul 18 01:15:04.554568 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.554547    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"
Jul 18 01:15:04.560766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.560745    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"]
Jul 18 01:15:04.732940 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.732854    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef96a4e-e5c0-4004-a6a5-9d477944d20e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-xw6rx\" (UID: \"eef96a4e-e5c0-4004-a6a5-9d477944d20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"
Jul 18 01:15:04.834015 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.833983    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef96a4e-e5c0-4004-a6a5-9d477944d20e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-xw6rx\" (UID: \"eef96a4e-e5c0-4004-a6a5-9d477944d20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"
Jul 18 01:15:04.834591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.834574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef96a4e-e5c0-4004-a6a5-9d477944d20e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-xw6rx\" (UID: \"eef96a4e-e5c0-4004-a6a5-9d477944d20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"
Jul 18 01:15:04.866897 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.866869    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"
Jul 18 01:15:04.999055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:04.999029    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"]
Jul 18 01:15:05.000681 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:15:05.000656    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef96a4e_e5c0_4004_a6a5_9d477944d20e.slice/crio-84ad1ccbc47c22be0454c2696710fadd87d4155a21a83cf06f7cc62518e2f305 WatchSource:0}: Error finding container 84ad1ccbc47c22be0454c2696710fadd87d4155a21a83cf06f7cc62518e2f305: Status 404 returned error can't find the container with id 84ad1ccbc47c22be0454c2696710fadd87d4155a21a83cf06f7cc62518e2f305
Jul 18 01:15:05.413309 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:05.413273    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" event={"ID":"eef96a4e-e5c0-4004-a6a5-9d477944d20e","Type":"ContainerStarted","Data":"885b889dc92df0553697a17970e55e1858ef2c6b19b5d04c67b9ce426b537d26"}
Jul 18 01:15:05.413710 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:05.413329    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" event={"ID":"eef96a4e-e5c0-4004-a6a5-9d477944d20e","Type":"ContainerStarted","Data":"84ad1ccbc47c22be0454c2696710fadd87d4155a21a83cf06f7cc62518e2f305"}
Jul 18 01:15:05.429063 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:05.429011    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" podStartSLOduration=1.428996057 podStartE2EDuration="1.428996057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:15:05.426826651 +0000 UTC m=+2061.552761632" watchObservedRunningTime="2026-07-18 01:15:05.428996057 +0000 UTC m=+2061.554931036"
Jul 18 01:15:06.419759 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.419722    2554 generic.go:356] "Generic (PLEG): container finished" podID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerID="885b889dc92df0553697a17970e55e1858ef2c6b19b5d04c67b9ce426b537d26" exitCode=7
Jul 18 01:15:06.420165 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.419811    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" event={"ID":"eef96a4e-e5c0-4004-a6a5-9d477944d20e","Type":"ContainerDied","Data":"885b889dc92df0553697a17970e55e1858ef2c6b19b5d04c67b9ce426b537d26"}
Jul 18 01:15:06.420165 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.419984    2554 scope.go:117] "RemoveContainer" containerID="885b889dc92df0553697a17970e55e1858ef2c6b19b5d04c67b9ce426b537d26"
Jul 18 01:15:06.549270 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.549238    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"]
Jul 18 01:15:06.553751 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.553731    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"
Jul 18 01:15:06.560633 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.560591    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"]
Jul 18 01:15:06.653449 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.653422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d170585-bd39-4b5c-b2dd-6098a48e9b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg\" (UID: \"4d170585-bd39-4b5c-b2dd-6098a48e9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"
Jul 18 01:15:06.754608 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.754578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d170585-bd39-4b5c-b2dd-6098a48e9b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg\" (UID: \"4d170585-bd39-4b5c-b2dd-6098a48e9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"
Jul 18 01:15:06.755183 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.755165    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d170585-bd39-4b5c-b2dd-6098a48e9b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg\" (UID: \"4d170585-bd39-4b5c-b2dd-6098a48e9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"
Jul 18 01:15:06.865871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.865836    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"
Jul 18 01:15:06.998085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:06.998061    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"]
Jul 18 01:15:07.000719 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:15:07.000691    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d170585_bd39_4b5c_b2dd_6098a48e9b0e.slice/crio-0eb5e1fc924ae37c753a5ef1d1f45550f55fd53b467db028f63254a128bbfcb8 WatchSource:0}: Error finding container 0eb5e1fc924ae37c753a5ef1d1f45550f55fd53b467db028f63254a128bbfcb8: Status 404 returned error can't find the container with id 0eb5e1fc924ae37c753a5ef1d1f45550f55fd53b467db028f63254a128bbfcb8
Jul 18 01:15:07.426644 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:07.426542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" event={"ID":"4d170585-bd39-4b5c-b2dd-6098a48e9b0e","Type":"ContainerStarted","Data":"8315aecf1fbce949572699f61f48f3ce8a036b5c25b13f4e3bf198cb199df424"}
Jul 18 01:15:07.426644 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:07.426599    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" event={"ID":"4d170585-bd39-4b5c-b2dd-6098a48e9b0e","Type":"ContainerStarted","Data":"0eb5e1fc924ae37c753a5ef1d1f45550f55fd53b467db028f63254a128bbfcb8"}
Jul 18 01:15:07.428434 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:07.428409    2554 generic.go:356] "Generic (PLEG): container finished" podID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerID="943df0480510bb1e9b11c2a481c7c8d116423cd3dc176652de5227c2c7d90e03" exitCode=7
Jul 18 01:15:07.428545 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:07.428440    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" event={"ID":"eef96a4e-e5c0-4004-a6a5-9d477944d20e","Type":"ContainerDied","Data":"943df0480510bb1e9b11c2a481c7c8d116423cd3dc176652de5227c2c7d90e03"}
Jul 18 01:15:07.428545 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:07.428477    2554 scope.go:117] "RemoveContainer" containerID="885b889dc92df0553697a17970e55e1858ef2c6b19b5d04c67b9ce426b537d26"
Jul 18 01:15:07.428702 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:07.428686    2554 scope.go:117] "RemoveContainer" containerID="943df0480510bb1e9b11c2a481c7c8d116423cd3dc176652de5227c2c7d90e03"
Jul 18 01:15:07.428926 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:07.428903    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-xw6rx_odh-ai-gateway-infra(eef96a4e-e5c0-4004-a6a5-9d477944d20e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e"
Jul 18 01:15:07.442376 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:07.442331    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" podStartSLOduration=1.442314751 podStartE2EDuration="1.442314751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:15:07.441350296 +0000 UTC m=+2063.567285275" watchObservedRunningTime="2026-07-18 01:15:07.442314751 +0000 UTC m=+2063.568249755"
Jul 18 01:15:08.434502 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:08.434471    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerID="8315aecf1fbce949572699f61f48f3ce8a036b5c25b13f4e3bf198cb199df424" exitCode=6
Jul 18 01:15:08.434974 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:08.434564    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" event={"ID":"4d170585-bd39-4b5c-b2dd-6098a48e9b0e","Type":"ContainerDied","Data":"8315aecf1fbce949572699f61f48f3ce8a036b5c25b13f4e3bf198cb199df424"}
Jul 18 01:15:08.434974 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:08.434759    2554 scope.go:117] "RemoveContainer" containerID="8315aecf1fbce949572699f61f48f3ce8a036b5c25b13f4e3bf198cb199df424"
Jul 18 01:15:08.436303 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:08.436285    2554 scope.go:117] "RemoveContainer" containerID="943df0480510bb1e9b11c2a481c7c8d116423cd3dc176652de5227c2c7d90e03"
Jul 18 01:15:08.436484 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:08.436467    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-xw6rx_odh-ai-gateway-infra(eef96a4e-e5c0-4004-a6a5-9d477944d20e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e"
Jul 18 01:15:09.443027 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:09.442927    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerID="c5dff4e9d1e715cb97b45148c8a5ec0c3ed0d95a055ec9dc2f5d78857583b86e" exitCode=6
Jul 18 01:15:09.443027 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:09.442967    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" event={"ID":"4d170585-bd39-4b5c-b2dd-6098a48e9b0e","Type":"ContainerDied","Data":"c5dff4e9d1e715cb97b45148c8a5ec0c3ed0d95a055ec9dc2f5d78857583b86e"}
Jul 18 01:15:09.443027 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:09.443027    2554 scope.go:117] "RemoveContainer" containerID="8315aecf1fbce949572699f61f48f3ce8a036b5c25b13f4e3bf198cb199df424"
Jul 18 01:15:09.443592 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:09.443218    2554 scope.go:117] "RemoveContainer" containerID="c5dff4e9d1e715cb97b45148c8a5ec0c3ed0d95a055ec9dc2f5d78857583b86e"
Jul 18 01:15:09.443592 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:09.443469    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg_odh-ai-gateway-infra(4d170585-bd39-4b5c-b2dd-6098a48e9b0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e"
Jul 18 01:15:10.449392 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:10.449366    2554 scope.go:117] "RemoveContainer" containerID="c5dff4e9d1e715cb97b45148c8a5ec0c3ed0d95a055ec9dc2f5d78857583b86e"
Jul 18 01:15:10.449827 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:10.449577    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg_odh-ai-gateway-infra(4d170585-bd39-4b5c-b2dd-6098a48e9b0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e"
Jul 18 01:15:13.330830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:13.330795    2554 scope.go:117] "RemoveContainer" containerID="2463499dd74c3d8aa4f7617472326f49797f0e77d531c031401719a053d7d54f"
Jul 18 01:15:14.471726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:14.471690    2554 generic.go:356] "Generic (PLEG): container finished" podID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerID="97545ba4522347f425733ea9bede3219e391541eb8e3a7963b4f5fe6436ad9aa" exitCode=6
Jul 18 01:15:14.472190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:14.471742    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" event={"ID":"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6","Type":"ContainerDied","Data":"97545ba4522347f425733ea9bede3219e391541eb8e3a7963b4f5fe6436ad9aa"}
Jul 18 01:15:14.472190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:14.471796    2554 scope.go:117] "RemoveContainer" containerID="2463499dd74c3d8aa4f7617472326f49797f0e77d531c031401719a053d7d54f"
Jul 18 01:15:14.472190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:14.472065    2554 scope.go:117] "RemoveContainer" containerID="97545ba4522347f425733ea9bede3219e391541eb8e3a7963b4f5fe6436ad9aa"
Jul 18 01:15:14.472330 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:14.472300    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8_odh-ai-gateway-infra(5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6"
Jul 18 01:15:15.496386 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:15.496350    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"]
Jul 18 01:15:15.629976 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:15.629950    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"
Jul 18 01:15:15.741776 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:15.741734    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6-maas-api-service-ca\") pod \"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6\" (UID: \"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6\") "
Jul 18 01:15:15.742080 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:15.742057    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" (UID: "5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:15:15.843211 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:15.843133    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:15:16.330960 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:16.330927    2554 scope.go:117] "RemoveContainer" containerID="c8b937b1904aa4795ad8ff588c88f780760042fc5329ae564440d0e1317d14d2"
Jul 18 01:15:16.485661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:16.485630    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"
Jul 18 01:15:16.485661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:16.485653    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8" event={"ID":"5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6","Type":"ContainerDied","Data":"2c62b7a1294655e10bd891b1c029a5a7908b583b3fef63240a2ec245989ef616"}
Jul 18 01:15:16.485879 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:16.485718    2554 scope.go:117] "RemoveContainer" containerID="97545ba4522347f425733ea9bede3219e391541eb8e3a7963b4f5fe6436ad9aa"
Jul 18 01:15:16.502221 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:16.502193    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"]
Jul 18 01:15:16.507686 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:16.507665    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jcdw8"]
Jul 18 01:15:17.494018 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:17.493980    2554 generic.go:356] "Generic (PLEG): container finished" podID="031ca428-9136-4309-b972-e8529e376ad8" containerID="e0e0766050bbb55457ad1a428933ba97181df25682908aa56fda89aaaa173466" exitCode=7
Jul 18 01:15:17.494184 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:17.494060    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" event={"ID":"031ca428-9136-4309-b972-e8529e376ad8","Type":"ContainerDied","Data":"e0e0766050bbb55457ad1a428933ba97181df25682908aa56fda89aaaa173466"}
Jul 18 01:15:17.494184 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:17.494134    2554 scope.go:117] "RemoveContainer" containerID="c8b937b1904aa4795ad8ff588c88f780760042fc5329ae564440d0e1317d14d2"
Jul 18 01:15:17.494409 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:17.494390    2554 scope.go:117] "RemoveContainer" containerID="e0e0766050bbb55457ad1a428933ba97181df25682908aa56fda89aaaa173466"
Jul 18 01:15:17.494744 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:17.494716    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd_odh-ai-gateway-infra(031ca428-9136-4309-b972-e8529e376ad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" podUID="031ca428-9136-4309-b972-e8529e376ad8"
Jul 18 01:15:18.336266 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:18.336234    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" path="/var/lib/kubelet/pods/5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6/volumes"
Jul 18 01:15:18.519477 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:18.519440    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"]
Jul 18 01:15:18.655741 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:18.655714    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"
Jul 18 01:15:19.512057 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:19.512026    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"
Jul 18 01:15:19.512543 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:19.512061    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd" event={"ID":"031ca428-9136-4309-b972-e8529e376ad8","Type":"ContainerDied","Data":"cda0b846901ead171541b5e95279e6a5a93b02a809f71bbd9119665aecac978a"}
Jul 18 01:15:19.512543 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:19.512120    2554 scope.go:117] "RemoveContainer" containerID="e0e0766050bbb55457ad1a428933ba97181df25682908aa56fda89aaaa173466"
Jul 18 01:15:19.535385 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:19.535358    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"]
Jul 18 01:15:19.541163 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:19.541134    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4701d191-29738955-sznsd"]
Jul 18 01:15:20.336375 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:20.336342    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="031ca428-9136-4309-b972-e8529e376ad8" path="/var/lib/kubelet/pods/031ca428-9136-4309-b972-e8529e376ad8/volumes"
Jul 18 01:15:23.330681 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:23.330604    2554 scope.go:117] "RemoveContainer" containerID="943df0480510bb1e9b11c2a481c7c8d116423cd3dc176652de5227c2c7d90e03"
Jul 18 01:15:23.535276 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:23.535245    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" event={"ID":"eef96a4e-e5c0-4004-a6a5-9d477944d20e","Type":"ContainerStarted","Data":"a0652f49a0dbf3436063104cd9baf3f9b3b8ac302ee3d46a20d0ca03d435889b"}
Jul 18 01:15:24.541513 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:24.541482    2554 generic.go:356] "Generic (PLEG): container finished" podID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerID="a0652f49a0dbf3436063104cd9baf3f9b3b8ac302ee3d46a20d0ca03d435889b" exitCode=7
Jul 18 01:15:24.541954 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:24.541555    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" event={"ID":"eef96a4e-e5c0-4004-a6a5-9d477944d20e","Type":"ContainerDied","Data":"a0652f49a0dbf3436063104cd9baf3f9b3b8ac302ee3d46a20d0ca03d435889b"}
Jul 18 01:15:24.541954 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:24.541644    2554 scope.go:117] "RemoveContainer" containerID="943df0480510bb1e9b11c2a481c7c8d116423cd3dc176652de5227c2c7d90e03"
Jul 18 01:15:24.541954 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:24.541937    2554 scope.go:117] "RemoveContainer" containerID="a0652f49a0dbf3436063104cd9baf3f9b3b8ac302ee3d46a20d0ca03d435889b"
Jul 18 01:15:24.542174 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:24.542153    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-xw6rx_odh-ai-gateway-infra(eef96a4e-e5c0-4004-a6a5-9d477944d20e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e"
Jul 18 01:15:24.581430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:24.581402    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"]
Jul 18 01:15:25.330204 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:25.330174    2554 scope.go:117] "RemoveContainer" containerID="c5dff4e9d1e715cb97b45148c8a5ec0c3ed0d95a055ec9dc2f5d78857583b86e"
Jul 18 01:15:25.548568 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:25.548536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" event={"ID":"4d170585-bd39-4b5c-b2dd-6098a48e9b0e","Type":"ContainerStarted","Data":"5b8600f7ba49672369635de09594987ef706e605e4caef1ed659f338fb8deabb"}
Jul 18 01:15:25.692656 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:25.692609    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"
Jul 18 01:15:25.745034 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:25.744999    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef96a4e-e5c0-4004-a6a5-9d477944d20e-maas-api-service-ca\") pod \"eef96a4e-e5c0-4004-a6a5-9d477944d20e\" (UID: \"eef96a4e-e5c0-4004-a6a5-9d477944d20e\") "
Jul 18 01:15:25.745363 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:25.745340    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef96a4e-e5c0-4004-a6a5-9d477944d20e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eef96a4e-e5c0-4004-a6a5-9d477944d20e" (UID: "eef96a4e-e5c0-4004-a6a5-9d477944d20e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:15:25.846329 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:25.846289    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef96a4e-e5c0-4004-a6a5-9d477944d20e-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:15:26.557190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.557155    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerID="5b8600f7ba49672369635de09594987ef706e605e4caef1ed659f338fb8deabb" exitCode=6
Jul 18 01:15:26.557655 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.557223    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" event={"ID":"4d170585-bd39-4b5c-b2dd-6098a48e9b0e","Type":"ContainerDied","Data":"5b8600f7ba49672369635de09594987ef706e605e4caef1ed659f338fb8deabb"}
Jul 18 01:15:26.557655 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.557290    2554 scope.go:117] "RemoveContainer" containerID="c5dff4e9d1e715cb97b45148c8a5ec0c3ed0d95a055ec9dc2f5d78857583b86e"
Jul 18 01:15:26.557655 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.557645    2554 scope.go:117] "RemoveContainer" containerID="5b8600f7ba49672369635de09594987ef706e605e4caef1ed659f338fb8deabb"
Jul 18 01:15:26.557906 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:26.557884    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg_odh-ai-gateway-infra(4d170585-bd39-4b5c-b2dd-6098a48e9b0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e"
Jul 18 01:15:26.558667 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.558643    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx" event={"ID":"eef96a4e-e5c0-4004-a6a5-9d477944d20e","Type":"ContainerDied","Data":"84ad1ccbc47c22be0454c2696710fadd87d4155a21a83cf06f7cc62518e2f305"}
Jul 18 01:15:26.558734 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.558660    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"
Jul 18 01:15:26.568530 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.568512    2554 scope.go:117] "RemoveContainer" containerID="a0652f49a0dbf3436063104cd9baf3f9b3b8ac302ee3d46a20d0ca03d435889b"
Jul 18 01:15:26.573310 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.573284    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"]
Jul 18 01:15:26.597595 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.597520    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"]
Jul 18 01:15:26.602474 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:26.602449    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-xw6rx"]
Jul 18 01:15:27.704501 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:27.704478    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"
Jul 18 01:15:27.762987 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:27.762953    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d170585-bd39-4b5c-b2dd-6098a48e9b0e-maas-api-service-ca\") pod \"4d170585-bd39-4b5c-b2dd-6098a48e9b0e\" (UID: \"4d170585-bd39-4b5c-b2dd-6098a48e9b0e\") "
Jul 18 01:15:27.763289 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:27.763267    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d170585-bd39-4b5c-b2dd-6098a48e9b0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d170585-bd39-4b5c-b2dd-6098a48e9b0e" (UID: "4d170585-bd39-4b5c-b2dd-6098a48e9b0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:15:27.864431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:27.864350    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d170585-bd39-4b5c-b2dd-6098a48e9b0e-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:15:28.335958 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:28.335924    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" path="/var/lib/kubelet/pods/eef96a4e-e5c0-4004-a6a5-9d477944d20e/volumes"
Jul 18 01:15:28.571955 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:28.571928    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"
Jul 18 01:15:28.572126 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:28.571927    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg" event={"ID":"4d170585-bd39-4b5c-b2dd-6098a48e9b0e","Type":"ContainerDied","Data":"0eb5e1fc924ae37c753a5ef1d1f45550f55fd53b467db028f63254a128bbfcb8"}
Jul 18 01:15:28.572126 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:28.572076    2554 scope.go:117] "RemoveContainer" containerID="5b8600f7ba49672369635de09594987ef706e605e4caef1ed659f338fb8deabb"
Jul 18 01:15:28.587012 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:28.586951    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"]
Jul 18 01:15:28.594791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:28.594764    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-nwpkg"]
Jul 18 01:15:30.336369 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:30.336336    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" path="/var/lib/kubelet/pods/4d170585-bd39-4b5c-b2dd-6098a48e9b0e/volumes"
Jul 18 01:15:48.566289 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566253    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"]
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566654    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566667    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566686    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566692    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566698    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566705    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566714    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566719    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566726    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:15:48.566726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566731    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566740    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566745    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566752    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566758    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566764    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566769    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566774    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566779    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566785    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566790    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566843    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566849    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566855    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566860    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566866    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566872    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566878    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:15:48.567084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.566884    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:15:48.569831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.569815    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"
Jul 18 01:15:48.572279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.572257    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:15:48.582824 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.582516    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170b0667-b8f6-41a7-b341-5d11392b523a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-8wplk\" (UID: \"170b0667-b8f6-41a7-b341-5d11392b523a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"
Jul 18 01:15:48.586894 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.586866    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"]
Jul 18 01:15:48.683843 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.683799    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170b0667-b8f6-41a7-b341-5d11392b523a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-8wplk\" (UID: \"170b0667-b8f6-41a7-b341-5d11392b523a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"
Jul 18 01:15:48.684418 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.684397    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170b0667-b8f6-41a7-b341-5d11392b523a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-8wplk\" (UID: \"170b0667-b8f6-41a7-b341-5d11392b523a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"
Jul 18 01:15:48.897066 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:48.896978    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"
Jul 18 01:15:49.030410 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:49.030312    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"]
Jul 18 01:15:49.032780 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:15:49.032748    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170b0667_b8f6_41a7_b341_5d11392b523a.slice/crio-a14b2227129adedf60be09f98e1c9ebd3ad0c54b4b7151c549bf740cd9dba3ac WatchSource:0}: Error finding container a14b2227129adedf60be09f98e1c9ebd3ad0c54b4b7151c549bf740cd9dba3ac: Status 404 returned error can't find the container with id a14b2227129adedf60be09f98e1c9ebd3ad0c54b4b7151c549bf740cd9dba3ac
Jul 18 01:15:49.680791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:49.680757    2554 generic.go:356] "Generic (PLEG): container finished" podID="170b0667-b8f6-41a7-b341-5d11392b523a" containerID="cb20d14e92a099fe7fc332705713db43eb4ec9f35d3fad626fa57353eb4c6704" exitCode=6
Jul 18 01:15:49.681140 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:49.680842    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" event={"ID":"170b0667-b8f6-41a7-b341-5d11392b523a","Type":"ContainerDied","Data":"cb20d14e92a099fe7fc332705713db43eb4ec9f35d3fad626fa57353eb4c6704"}
Jul 18 01:15:49.681140 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:49.680882    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" event={"ID":"170b0667-b8f6-41a7-b341-5d11392b523a","Type":"ContainerStarted","Data":"a14b2227129adedf60be09f98e1c9ebd3ad0c54b4b7151c549bf740cd9dba3ac"}
Jul 18 01:15:49.681140 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:49.680989    2554 scope.go:117] "RemoveContainer" containerID="cb20d14e92a099fe7fc332705713db43eb4ec9f35d3fad626fa57353eb4c6704"
Jul 18 01:15:50.686823 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:50.686741    2554 generic.go:356] "Generic (PLEG): container finished" podID="170b0667-b8f6-41a7-b341-5d11392b523a" containerID="3c3fce17a3db0bdbaa023950f15d90b49d6847e66eb3c34211eb14307c32cf63" exitCode=6
Jul 18 01:15:50.687190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:50.686825    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" event={"ID":"170b0667-b8f6-41a7-b341-5d11392b523a","Type":"ContainerDied","Data":"3c3fce17a3db0bdbaa023950f15d90b49d6847e66eb3c34211eb14307c32cf63"}
Jul 18 01:15:50.687190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:50.686890    2554 scope.go:117] "RemoveContainer" containerID="cb20d14e92a099fe7fc332705713db43eb4ec9f35d3fad626fa57353eb4c6704"
Jul 18 01:15:50.687190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:50.687111    2554 scope.go:117] "RemoveContainer" containerID="3c3fce17a3db0bdbaa023950f15d90b49d6847e66eb3c34211eb14307c32cf63"
Jul 18 01:15:50.687387 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:50.687369    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-8wplk_odh-ai-gateway-infra(170b0667-b8f6-41a7-b341-5d11392b523a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" podUID="170b0667-b8f6-41a7-b341-5d11392b523a"
Jul 18 01:15:51.694018 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:15:51.693990    2554 scope.go:117] "RemoveContainer" containerID="3c3fce17a3db0bdbaa023950f15d90b49d6847e66eb3c34211eb14307c32cf63"
Jul 18 01:15:51.694378 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:15:51.694274    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-8wplk_odh-ai-gateway-infra(170b0667-b8f6-41a7-b341-5d11392b523a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" podUID="170b0667-b8f6-41a7-b341-5d11392b523a"
Jul 18 01:16:02.330459 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:02.330427    2554 scope.go:117] "RemoveContainer" containerID="3c3fce17a3db0bdbaa023950f15d90b49d6847e66eb3c34211eb14307c32cf63"
Jul 18 01:16:02.750399 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:02.750359    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" event={"ID":"170b0667-b8f6-41a7-b341-5d11392b523a","Type":"ContainerStarted","Data":"9a156720e39fad14b65b12155d6f9e44fb3cae537fbeb202bfab0409774f444c"}
Jul 18 01:16:02.765315 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:02.765262    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" podStartSLOduration=14.765246007 podStartE2EDuration="14.765246007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:16:02.764327235 +0000 UTC m=+2118.890262227" watchObservedRunningTime="2026-07-18 01:16:02.765246007 +0000 UTC m=+2118.891180987"
Jul 18 01:16:03.756493 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:03.756456    2554 generic.go:356] "Generic (PLEG): container finished" podID="170b0667-b8f6-41a7-b341-5d11392b523a" containerID="9a156720e39fad14b65b12155d6f9e44fb3cae537fbeb202bfab0409774f444c" exitCode=6
Jul 18 01:16:03.756493 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:03.756497    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" event={"ID":"170b0667-b8f6-41a7-b341-5d11392b523a","Type":"ContainerDied","Data":"9a156720e39fad14b65b12155d6f9e44fb3cae537fbeb202bfab0409774f444c"}
Jul 18 01:16:03.757066 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:03.756546    2554 scope.go:117] "RemoveContainer" containerID="3c3fce17a3db0bdbaa023950f15d90b49d6847e66eb3c34211eb14307c32cf63"
Jul 18 01:16:03.757066 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:03.756912    2554 scope.go:117] "RemoveContainer" containerID="9a156720e39fad14b65b12155d6f9e44fb3cae537fbeb202bfab0409774f444c"
Jul 18 01:16:03.757194 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:03.757170    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-8wplk_odh-ai-gateway-infra(170b0667-b8f6-41a7-b341-5d11392b523a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" podUID="170b0667-b8f6-41a7-b341-5d11392b523a"
Jul 18 01:16:03.775395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:03.775367    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"]
Jul 18 01:16:04.614223 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614192    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"]
Jul 18 01:16:04.614607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614596    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:16:04.614676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614609    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:16:04.614676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614637    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:16:04.614676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614642    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:16:04.614767 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614710    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef96a4e-e5c0-4004-a6a5-9d477944d20e" containerName="revoke-keys"
Jul 18 01:16:04.614767 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614721    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="031ca428-9136-4309-b972-e8529e376ad8" containerName="cleanup"
Jul 18 01:16:04.614767 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614731    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d170585-bd39-4b5c-b2dd-6098a48e9b0e" containerName="revoke-keys"
Jul 18 01:16:04.614767 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.614737    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5ce8d7-97fa-4289-bfb0-7f9d428ac4f6" containerName="revoke-keys"
Jul 18 01:16:04.619062 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.619046    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"
Jul 18 01:16:04.624896 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.624866    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"]
Jul 18 01:16:04.744680 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.744607    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e92e69-3a76-4f58-8d3a-c6b396bb0b5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-dckcs\" (UID: \"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"
Jul 18 01:16:04.845387 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.845344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e92e69-3a76-4f58-8d3a-c6b396bb0b5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-dckcs\" (UID: \"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"
Jul 18 01:16:04.846107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.846076    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e92e69-3a76-4f58-8d3a-c6b396bb0b5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-dckcs\" (UID: \"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"
Jul 18 01:16:04.905076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.905051    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"
Jul 18 01:16:04.932951 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:04.932923    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"
Jul 18 01:16:05.048237 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.048203    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170b0667-b8f6-41a7-b341-5d11392b523a-maas-api-service-ca\") pod \"170b0667-b8f6-41a7-b341-5d11392b523a\" (UID: \"170b0667-b8f6-41a7-b341-5d11392b523a\") "
Jul 18 01:16:05.048669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.048608    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170b0667-b8f6-41a7-b341-5d11392b523a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "170b0667-b8f6-41a7-b341-5d11392b523a" (UID: "170b0667-b8f6-41a7-b341-5d11392b523a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:16:05.061396 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.061368    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"]
Jul 18 01:16:05.063325 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:16:05.063293    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e92e69_3a76_4f58_8d3a_c6b396bb0b5d.slice/crio-ec137a9c2b5254666f160e0f4f3ee438b8a2d6e1d98bb603b9dc330e08f7dbd2 WatchSource:0}: Error finding container ec137a9c2b5254666f160e0f4f3ee438b8a2d6e1d98bb603b9dc330e08f7dbd2: Status 404 returned error can't find the container with id ec137a9c2b5254666f160e0f4f3ee438b8a2d6e1d98bb603b9dc330e08f7dbd2
Jul 18 01:16:05.149828 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.149767    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170b0667-b8f6-41a7-b341-5d11392b523a-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:16:05.775161 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.775133    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"
Jul 18 01:16:05.775367 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.775169    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk" event={"ID":"170b0667-b8f6-41a7-b341-5d11392b523a","Type":"ContainerDied","Data":"a14b2227129adedf60be09f98e1c9ebd3ad0c54b4b7151c549bf740cd9dba3ac"}
Jul 18 01:16:05.775367 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.775233    2554 scope.go:117] "RemoveContainer" containerID="9a156720e39fad14b65b12155d6f9e44fb3cae537fbeb202bfab0409774f444c"
Jul 18 01:16:05.777106 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.777068    2554 generic.go:356] "Generic (PLEG): container finished" podID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerID="8f41ef6c3e4989696e531a19b094a68f8ab60f7ced9d31242a654ba8cff5ff85" exitCode=7
Jul 18 01:16:05.777213 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.777132    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" event={"ID":"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d","Type":"ContainerDied","Data":"8f41ef6c3e4989696e531a19b094a68f8ab60f7ced9d31242a654ba8cff5ff85"}
Jul 18 01:16:05.777213 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.777179    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" event={"ID":"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d","Type":"ContainerStarted","Data":"ec137a9c2b5254666f160e0f4f3ee438b8a2d6e1d98bb603b9dc330e08f7dbd2"}
Jul 18 01:16:05.777362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.777349    2554 scope.go:117] "RemoveContainer" containerID="8f41ef6c3e4989696e531a19b094a68f8ab60f7ced9d31242a654ba8cff5ff85"
Jul 18 01:16:05.809813 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.809792    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"]
Jul 18 01:16:05.811552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:05.811534    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-8wplk"]
Jul 18 01:16:06.336395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.336352    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" path="/var/lib/kubelet/pods/170b0667-b8f6-41a7-b341-5d11392b523a/volumes"
Jul 18 01:16:06.606902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.606816    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"]
Jul 18 01:16:06.607235 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607222    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607236    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607247    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607252    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607260    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607266    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607330    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607337    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.607431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.607344    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="170b0667-b8f6-41a7-b341-5d11392b523a" containerName="revoke-keys"
Jul 18 01:16:06.610254 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.610237    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"
Jul 18 01:16:06.617834 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.617806    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"]
Jul 18 01:16:06.763844 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.763807    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn\" (UID: \"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"
Jul 18 01:16:06.785359 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.785319    2554 generic.go:356] "Generic (PLEG): container finished" podID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerID="805ac3c3258841c8a27dca7694946dff7a00010c592a421816ff67c2ff6d4e45" exitCode=7
Jul 18 01:16:06.785524 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.785357    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" event={"ID":"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d","Type":"ContainerDied","Data":"805ac3c3258841c8a27dca7694946dff7a00010c592a421816ff67c2ff6d4e45"}
Jul 18 01:16:06.785524 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.785413    2554 scope.go:117] "RemoveContainer" containerID="8f41ef6c3e4989696e531a19b094a68f8ab60f7ced9d31242a654ba8cff5ff85"
Jul 18 01:16:06.785661 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.785592    2554 scope.go:117] "RemoveContainer" containerID="805ac3c3258841c8a27dca7694946dff7a00010c592a421816ff67c2ff6d4e45"
Jul 18 01:16:06.785899 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:06.785875    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-dckcs_odh-ai-gateway-infra(79e92e69-3a76-4f58-8d3a-c6b396bb0b5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d"
Jul 18 01:16:06.865027 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.864955    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn\" (UID: \"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"
Jul 18 01:16:06.865542 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.865524    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn\" (UID: \"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"
Jul 18 01:16:06.922838 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:06.922811    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"
Jul 18 01:16:07.266367 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:07.266344    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"]
Jul 18 01:16:07.268784 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:16:07.268751    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33e90d0_f9f5_4d67_bb45_b52b9c3d3a48.slice/crio-2d24c1a6d409889acabb885ae806064e66e285fb6c74c7592509f9ad7b860b7e WatchSource:0}: Error finding container 2d24c1a6d409889acabb885ae806064e66e285fb6c74c7592509f9ad7b860b7e: Status 404 returned error can't find the container with id 2d24c1a6d409889acabb885ae806064e66e285fb6c74c7592509f9ad7b860b7e
Jul 18 01:16:07.792293 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:07.792189    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" event={"ID":"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48","Type":"ContainerStarted","Data":"af920764260f69ab8992a5723faba77eb20cb52ca41c92757c420f7570adcefe"}
Jul 18 01:16:07.792293 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:07.792257    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" event={"ID":"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48","Type":"ContainerStarted","Data":"2d24c1a6d409889acabb885ae806064e66e285fb6c74c7592509f9ad7b860b7e"}
Jul 18 01:16:07.793966 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:07.793948    2554 scope.go:117] "RemoveContainer" containerID="805ac3c3258841c8a27dca7694946dff7a00010c592a421816ff67c2ff6d4e45"
Jul 18 01:16:07.794136 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:07.794119    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-dckcs_odh-ai-gateway-infra(79e92e69-3a76-4f58-8d3a-c6b396bb0b5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d"
Jul 18 01:16:07.807882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:07.807836    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" podStartSLOduration=1.807822481 podStartE2EDuration="1.807822481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:16:07.806106508 +0000 UTC m=+2123.932041489" watchObservedRunningTime="2026-07-18 01:16:07.807822481 +0000 UTC m=+2123.933757459"
Jul 18 01:16:08.801345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:08.801310    2554 generic.go:356] "Generic (PLEG): container finished" podID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerID="af920764260f69ab8992a5723faba77eb20cb52ca41c92757c420f7570adcefe" exitCode=6
Jul 18 01:16:08.801790 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:08.801403    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" event={"ID":"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48","Type":"ContainerDied","Data":"af920764260f69ab8992a5723faba77eb20cb52ca41c92757c420f7570adcefe"}
Jul 18 01:16:08.801790 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:08.801538    2554 scope.go:117] "RemoveContainer" containerID="af920764260f69ab8992a5723faba77eb20cb52ca41c92757c420f7570adcefe"
Jul 18 01:16:09.808049 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:09.808017    2554 generic.go:356] "Generic (PLEG): container finished" podID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerID="5105e12e7187d370b1196b1f9f6639c18170a799d89f5ce6063e721e62bfc6e9" exitCode=6
Jul 18 01:16:09.808505 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:09.808096    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" event={"ID":"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48","Type":"ContainerDied","Data":"5105e12e7187d370b1196b1f9f6639c18170a799d89f5ce6063e721e62bfc6e9"}
Jul 18 01:16:09.808505 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:09.808153    2554 scope.go:117] "RemoveContainer" containerID="af920764260f69ab8992a5723faba77eb20cb52ca41c92757c420f7570adcefe"
Jul 18 01:16:09.808505 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:09.808280    2554 scope.go:117] "RemoveContainer" containerID="5105e12e7187d370b1196b1f9f6639c18170a799d89f5ce6063e721e62bfc6e9"
Jul 18 01:16:09.808628 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:09.808523    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn_odh-ai-gateway-infra(c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48"
Jul 18 01:16:10.814114 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:10.814086    2554 scope.go:117] "RemoveContainer" containerID="5105e12e7187d370b1196b1f9f6639c18170a799d89f5ce6063e721e62bfc6e9"
Jul 18 01:16:10.814482 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:10.814283    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn_odh-ai-gateway-infra(c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48"
Jul 18 01:16:20.331280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:20.331250    2554 scope.go:117] "RemoveContainer" containerID="805ac3c3258841c8a27dca7694946dff7a00010c592a421816ff67c2ff6d4e45"
Jul 18 01:16:20.867076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:20.866993    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" event={"ID":"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d","Type":"ContainerStarted","Data":"4a0fcc02e00eda94abb7091fa3086837a35c2c71bb751c498dfccc5d32161815"}
Jul 18 01:16:20.883468 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:20.883412    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" podStartSLOduration=16.883397108 podStartE2EDuration="16.883397108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:16:20.88188361 +0000 UTC m=+2137.007818586" watchObservedRunningTime="2026-07-18 01:16:20.883397108 +0000 UTC m=+2137.009332090"
Jul 18 01:16:21.874683 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:21.874641    2554 generic.go:356] "Generic (PLEG): container finished" podID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerID="4a0fcc02e00eda94abb7091fa3086837a35c2c71bb751c498dfccc5d32161815" exitCode=7
Jul 18 01:16:21.874683 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:21.874656    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" event={"ID":"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d","Type":"ContainerDied","Data":"4a0fcc02e00eda94abb7091fa3086837a35c2c71bb751c498dfccc5d32161815"}
Jul 18 01:16:21.875165 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:21.874764    2554 scope.go:117] "RemoveContainer" containerID="805ac3c3258841c8a27dca7694946dff7a00010c592a421816ff67c2ff6d4e45"
Jul 18 01:16:21.875165 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:21.875075    2554 scope.go:117] "RemoveContainer" containerID="4a0fcc02e00eda94abb7091fa3086837a35c2c71bb751c498dfccc5d32161815"
Jul 18 01:16:21.875337 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:21.875316    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-dckcs_odh-ai-gateway-infra(79e92e69-3a76-4f58-8d3a-c6b396bb0b5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d"
Jul 18 01:16:21.898129 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:21.898101    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"]
Jul 18 01:16:23.022055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.022031    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"
Jul 18 01:16:23.124961 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.124933    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e92e69-3a76-4f58-8d3a-c6b396bb0b5d-maas-api-service-ca\") pod \"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d\" (UID: \"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d\") "
Jul 18 01:16:23.125309 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.125283    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e92e69-3a76-4f58-8d3a-c6b396bb0b5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" (UID: "79e92e69-3a76-4f58-8d3a-c6b396bb0b5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:16:23.226647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.226536    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e92e69-3a76-4f58-8d3a-c6b396bb0b5d-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:16:23.331330 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.331297    2554 scope.go:117] "RemoveContainer" containerID="5105e12e7187d370b1196b1f9f6639c18170a799d89f5ce6063e721e62bfc6e9"
Jul 18 01:16:23.887960 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.887863    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"
Jul 18 01:16:23.887960 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.887901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs" event={"ID":"79e92e69-3a76-4f58-8d3a-c6b396bb0b5d","Type":"ContainerDied","Data":"ec137a9c2b5254666f160e0f4f3ee438b8a2d6e1d98bb603b9dc330e08f7dbd2"}
Jul 18 01:16:23.888204 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.887974    2554 scope.go:117] "RemoveContainer" containerID="4a0fcc02e00eda94abb7091fa3086837a35c2c71bb751c498dfccc5d32161815"
Jul 18 01:16:23.889877 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.889852    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" event={"ID":"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48","Type":"ContainerStarted","Data":"ae540177897a6eba8d14e0536a223e7f43f42f6b190e36951af838ed3c024746"}
Jul 18 01:16:23.922438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.922412    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"]
Jul 18 01:16:23.926009 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:23.925983    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-dckcs"]
Jul 18 01:16:24.337323 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:24.337291    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" path="/var/lib/kubelet/pods/79e92e69-3a76-4f58-8d3a-c6b396bb0b5d/volumes"
Jul 18 01:16:24.896948 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:24.896915    2554 generic.go:356] "Generic (PLEG): container finished" podID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerID="ae540177897a6eba8d14e0536a223e7f43f42f6b190e36951af838ed3c024746" exitCode=6
Jul 18 01:16:24.897109 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:24.896999    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" event={"ID":"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48","Type":"ContainerDied","Data":"ae540177897a6eba8d14e0536a223e7f43f42f6b190e36951af838ed3c024746"}
Jul 18 01:16:24.897109 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:24.897067    2554 scope.go:117] "RemoveContainer" containerID="5105e12e7187d370b1196b1f9f6639c18170a799d89f5ce6063e721e62bfc6e9"
Jul 18 01:16:24.897331 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:24.897310    2554 scope.go:117] "RemoveContainer" containerID="ae540177897a6eba8d14e0536a223e7f43f42f6b190e36951af838ed3c024746"
Jul 18 01:16:24.897610 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:24.897584    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn_odh-ai-gateway-infra(c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48"
Jul 18 01:16:24.919587 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:24.919556    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"]
Jul 18 01:16:26.041938 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.041908    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"
Jul 18 01:16:26.050384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.050359    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48-maas-api-service-ca\") pod \"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48\" (UID: \"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48\") "
Jul 18 01:16:26.050684 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.050663    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" (UID: "c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:16:26.151427 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.151385    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:16:26.909910 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.909882    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"
Jul 18 01:16:26.910142 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.909926    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn" event={"ID":"c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48","Type":"ContainerDied","Data":"2d24c1a6d409889acabb885ae806064e66e285fb6c74c7592509f9ad7b860b7e"}
Jul 18 01:16:26.910142 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.909987    2554 scope.go:117] "RemoveContainer" containerID="ae540177897a6eba8d14e0536a223e7f43f42f6b190e36951af838ed3c024746"
Jul 18 01:16:26.924295 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.924270    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"]
Jul 18 01:16:26.930020 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:26.929998    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ksrjn"]
Jul 18 01:16:28.336383 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:28.336350    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" path="/var/lib/kubelet/pods/c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48/volumes"
Jul 18 01:16:38.599416 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599330    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"]
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599748    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599761    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599781    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599786    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599792    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599800    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599810    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599815    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599822    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599827    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599880    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599888    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599893    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:38.599901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.599899    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:38.604972 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.604954    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"
Jul 18 01:16:38.607882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.607858    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:16:38.610069 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.610037    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"]
Jul 18 01:16:38.681028 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.680985    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c2a2a-c3df-4d43-b637-59f6be3ff989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-688z2\" (UID: \"2b2c2a2a-c3df-4d43-b637-59f6be3ff989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"
Jul 18 01:16:38.782442 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.782381    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c2a2a-c3df-4d43-b637-59f6be3ff989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-688z2\" (UID: \"2b2c2a2a-c3df-4d43-b637-59f6be3ff989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"
Jul 18 01:16:38.783055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.783032    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c2a2a-c3df-4d43-b637-59f6be3ff989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-688z2\" (UID: \"2b2c2a2a-c3df-4d43-b637-59f6be3ff989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"
Jul 18 01:16:38.918081 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:38.917995    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"
Jul 18 01:16:39.051018 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:39.050995    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"]
Jul 18 01:16:39.053546 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:16:39.053519    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2c2a2a_c3df_4d43_b637_59f6be3ff989.slice/crio-b74e261d2b955d2b77a143ed7976678b67bb7ff88b91ecb58092a27eaf9466a9 WatchSource:0}: Error finding container b74e261d2b955d2b77a143ed7976678b67bb7ff88b91ecb58092a27eaf9466a9: Status 404 returned error can't find the container with id b74e261d2b955d2b77a143ed7976678b67bb7ff88b91ecb58092a27eaf9466a9
Jul 18 01:16:39.986819 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:39.986725    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerID="49c65f9829feace51d3c0d1bd1dd387ba248d5d12939943a6b9fc045aa5b3e79" exitCode=6
Jul 18 01:16:39.986819 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:39.986808    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" event={"ID":"2b2c2a2a-c3df-4d43-b637-59f6be3ff989","Type":"ContainerDied","Data":"49c65f9829feace51d3c0d1bd1dd387ba248d5d12939943a6b9fc045aa5b3e79"}
Jul 18 01:16:39.987343 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:39.986858    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" event={"ID":"2b2c2a2a-c3df-4d43-b637-59f6be3ff989","Type":"ContainerStarted","Data":"b74e261d2b955d2b77a143ed7976678b67bb7ff88b91ecb58092a27eaf9466a9"}
Jul 18 01:16:39.987343 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:39.987068    2554 scope.go:117] "RemoveContainer" containerID="49c65f9829feace51d3c0d1bd1dd387ba248d5d12939943a6b9fc045aa5b3e79"
Jul 18 01:16:40.993298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:40.993262    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerID="dcc55100b71c819d0a56d34f9606c47e9eddd848dbe1aabd80c09f1875f24974" exitCode=6
Jul 18 01:16:40.993767 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:40.993349    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" event={"ID":"2b2c2a2a-c3df-4d43-b637-59f6be3ff989","Type":"ContainerDied","Data":"dcc55100b71c819d0a56d34f9606c47e9eddd848dbe1aabd80c09f1875f24974"}
Jul 18 01:16:40.993767 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:40.993430    2554 scope.go:117] "RemoveContainer" containerID="49c65f9829feace51d3c0d1bd1dd387ba248d5d12939943a6b9fc045aa5b3e79"
Jul 18 01:16:40.993767 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:40.993702    2554 scope.go:117] "RemoveContainer" containerID="dcc55100b71c819d0a56d34f9606c47e9eddd848dbe1aabd80c09f1875f24974"
Jul 18 01:16:40.993961 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:40.993941    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-688z2_odh-ai-gateway-infra(2b2c2a2a-c3df-4d43-b637-59f6be3ff989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989"
Jul 18 01:16:41.999998 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:41.999968    2554 scope.go:117] "RemoveContainer" containerID="dcc55100b71c819d0a56d34f9606c47e9eddd848dbe1aabd80c09f1875f24974"
Jul 18 01:16:42.000361 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:42.000196    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-688z2_odh-ai-gateway-infra(2b2c2a2a-c3df-4d43-b637-59f6be3ff989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989"
Jul 18 01:16:54.662909 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.662872    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"]
Jul 18 01:16:54.663389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.663289    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:54.663389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.663301    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:54.663389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.663368    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e92e69-3a76-4f58-8d3a-c6b396bb0b5d" containerName="revoke-keys"
Jul 18 01:16:54.663389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.663378    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33e90d0-f9f5-4d67-bb45-b52b9c3d3a48" containerName="revoke-keys"
Jul 18 01:16:54.666258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.666233    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"
Jul 18 01:16:54.678837 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.677319    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"]
Jul 18 01:16:54.736689 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.736606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5332b-4e51-443e-9b61-e9bb35faa510-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-wbbsq\" (UID: \"8cb5332b-4e51-443e-9b61-e9bb35faa510\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"
Jul 18 01:16:54.837897 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.837853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5332b-4e51-443e-9b61-e9bb35faa510-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-wbbsq\" (UID: \"8cb5332b-4e51-443e-9b61-e9bb35faa510\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"
Jul 18 01:16:54.838518 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.838495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5332b-4e51-443e-9b61-e9bb35faa510-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-wbbsq\" (UID: \"8cb5332b-4e51-443e-9b61-e9bb35faa510\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"
Jul 18 01:16:54.980591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:54.980493    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"
Jul 18 01:16:55.119307 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:55.119278    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"]
Jul 18 01:16:55.121272 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:16:55.121237    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb5332b_4e51_443e_9b61_e9bb35faa510.slice/crio-c34f7ce1db3cd422b7fd75679c40fe2cde51e0e4735b469e9803454dd41e1bd8 WatchSource:0}: Error finding container c34f7ce1db3cd422b7fd75679c40fe2cde51e0e4735b469e9803454dd41e1bd8: Status 404 returned error can't find the container with id c34f7ce1db3cd422b7fd75679c40fe2cde51e0e4735b469e9803454dd41e1bd8
Jul 18 01:16:55.330768 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:55.330744    2554 scope.go:117] "RemoveContainer" containerID="dcc55100b71c819d0a56d34f9606c47e9eddd848dbe1aabd80c09f1875f24974"
Jul 18 01:16:56.071370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.071336    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerID="7d790c6b372b2ad2d4deca717a56efc0f7f976d31710bf4698ab662701c811f9" exitCode=6
Jul 18 01:16:56.071855 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.071409    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" event={"ID":"2b2c2a2a-c3df-4d43-b637-59f6be3ff989","Type":"ContainerDied","Data":"7d790c6b372b2ad2d4deca717a56efc0f7f976d31710bf4698ab662701c811f9"}
Jul 18 01:16:56.071855 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.071479    2554 scope.go:117] "RemoveContainer" containerID="dcc55100b71c819d0a56d34f9606c47e9eddd848dbe1aabd80c09f1875f24974"
Jul 18 01:16:56.071960 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.071867    2554 scope.go:117] "RemoveContainer" containerID="7d790c6b372b2ad2d4deca717a56efc0f7f976d31710bf4698ab662701c811f9"
Jul 18 01:16:56.072141 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:56.072117    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-688z2_odh-ai-gateway-infra(2b2c2a2a-c3df-4d43-b637-59f6be3ff989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989"
Jul 18 01:16:56.073084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.073063    2554 generic.go:356] "Generic (PLEG): container finished" podID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerID="fa76761ec99508320e099647bb9857fdf8ab4426f96544f1698a55ff038f3a88" exitCode=7
Jul 18 01:16:56.073162 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.073130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" event={"ID":"8cb5332b-4e51-443e-9b61-e9bb35faa510","Type":"ContainerDied","Data":"fa76761ec99508320e099647bb9857fdf8ab4426f96544f1698a55ff038f3a88"}
Jul 18 01:16:56.073208 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.073177    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" event={"ID":"8cb5332b-4e51-443e-9b61-e9bb35faa510","Type":"ContainerStarted","Data":"c34f7ce1db3cd422b7fd75679c40fe2cde51e0e4735b469e9803454dd41e1bd8"}
Jul 18 01:16:56.073283 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:56.073271    2554 scope.go:117] "RemoveContainer" containerID="fa76761ec99508320e099647bb9857fdf8ab4426f96544f1698a55ff038f3a88"
Jul 18 01:16:57.079453 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.079423    2554 generic.go:356] "Generic (PLEG): container finished" podID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerID="6640f5f1149f3858627adebc242f0bbe09425e0a0066a63d7cc8c1b5e4dbeb4b" exitCode=7
Jul 18 01:16:57.079990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.079519    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" event={"ID":"8cb5332b-4e51-443e-9b61-e9bb35faa510","Type":"ContainerDied","Data":"6640f5f1149f3858627adebc242f0bbe09425e0a0066a63d7cc8c1b5e4dbeb4b"}
Jul 18 01:16:57.079990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.079596    2554 scope.go:117] "RemoveContainer" containerID="fa76761ec99508320e099647bb9857fdf8ab4426f96544f1698a55ff038f3a88"
Jul 18 01:16:57.079990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.079719    2554 scope.go:117] "RemoveContainer" containerID="6640f5f1149f3858627adebc242f0bbe09425e0a0066a63d7cc8c1b5e4dbeb4b"
Jul 18 01:16:57.079990 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:57.079974    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-wbbsq_odh-ai-gateway-infra(8cb5332b-4e51-443e-9b61-e9bb35faa510)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510"
Jul 18 01:16:57.098804 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.098782    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"]
Jul 18 01:16:57.234796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.234770    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"
Jul 18 01:16:57.361761 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.361670    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c2a2a-c3df-4d43-b637-59f6be3ff989-maas-api-service-ca\") pod \"2b2c2a2a-c3df-4d43-b637-59f6be3ff989\" (UID: \"2b2c2a2a-c3df-4d43-b637-59f6be3ff989\") "
Jul 18 01:16:57.362026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.362003    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2c2a2a-c3df-4d43-b637-59f6be3ff989-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b2c2a2a-c3df-4d43-b637-59f6be3ff989" (UID: "2b2c2a2a-c3df-4d43-b637-59f6be3ff989"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:16:57.463395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:57.463357    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2c2a2a-c3df-4d43-b637-59f6be3ff989-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:16:58.087440 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:58.087414    2554 scope.go:117] "RemoveContainer" containerID="6640f5f1149f3858627adebc242f0bbe09425e0a0066a63d7cc8c1b5e4dbeb4b"
Jul 18 01:16:58.087951 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:16:58.087671    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-wbbsq_odh-ai-gateway-infra(8cb5332b-4e51-443e-9b61-e9bb35faa510)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510"
Jul 18 01:16:58.088610 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:58.088589    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2" event={"ID":"2b2c2a2a-c3df-4d43-b637-59f6be3ff989","Type":"ContainerDied","Data":"b74e261d2b955d2b77a143ed7976678b67bb7ff88b91ecb58092a27eaf9466a9"}
Jul 18 01:16:58.088687 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:58.088631    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"
Jul 18 01:16:58.088687 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:58.088665    2554 scope.go:117] "RemoveContainer" containerID="7d790c6b372b2ad2d4deca717a56efc0f7f976d31710bf4698ab662701c811f9"
Jul 18 01:16:58.115769 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:58.115734    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"]
Jul 18 01:16:58.119179 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:58.119153    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-688z2"]
Jul 18 01:16:58.336458 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:16:58.336422    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" path="/var/lib/kubelet/pods/2b2c2a2a-c3df-4d43-b637-59f6be3ff989/volumes"
Jul 18 01:17:06.650770 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.650729    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"]
Jul 18 01:17:06.651250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.651143    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:06.651250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.651154    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:06.651250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.651177    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:06.651250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.651183    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:06.651250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.651239    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:06.651250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.651252    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:06.654090 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.654071    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"
Jul 18 01:17:06.660494 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.660466    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"]
Jul 18 01:17:06.753736 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.753690    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa64cd-b809-490a-9848-e78d0bc3052d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn\" (UID: \"2ffa64cd-b809-490a-9848-e78d0bc3052d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"
Jul 18 01:17:06.854808 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.854767    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa64cd-b809-490a-9848-e78d0bc3052d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn\" (UID: \"2ffa64cd-b809-490a-9848-e78d0bc3052d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"
Jul 18 01:17:06.855401 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.855379    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa64cd-b809-490a-9848-e78d0bc3052d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn\" (UID: \"2ffa64cd-b809-490a-9848-e78d0bc3052d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"
Jul 18 01:17:06.967782 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:06.967694    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"
Jul 18 01:17:07.103383 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:07.103353    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"]
Jul 18 01:17:07.105297 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:17:07.105271    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffa64cd_b809_490a_9848_e78d0bc3052d.slice/crio-b47db9c2bc03ed742455b59e98b97286b4427a3e4089c43d6661de01cf311441 WatchSource:0}: Error finding container b47db9c2bc03ed742455b59e98b97286b4427a3e4089c43d6661de01cf311441: Status 404 returned error can't find the container with id b47db9c2bc03ed742455b59e98b97286b4427a3e4089c43d6661de01cf311441
Jul 18 01:17:07.139084 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:07.138889    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" event={"ID":"2ffa64cd-b809-490a-9848-e78d0bc3052d","Type":"ContainerStarted","Data":"b47db9c2bc03ed742455b59e98b97286b4427a3e4089c43d6661de01cf311441"}
Jul 18 01:17:08.145377 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:08.145344    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerID="3fcc4ded7363ce0f4a4a2021968139c2aaefccbf674239ab072181cfca20c6bc" exitCode=6
Jul 18 01:17:08.145795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:08.145417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" event={"ID":"2ffa64cd-b809-490a-9848-e78d0bc3052d","Type":"ContainerDied","Data":"3fcc4ded7363ce0f4a4a2021968139c2aaefccbf674239ab072181cfca20c6bc"}
Jul 18 01:17:08.145795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:08.145573    2554 scope.go:117] "RemoveContainer" containerID="3fcc4ded7363ce0f4a4a2021968139c2aaefccbf674239ab072181cfca20c6bc"
Jul 18 01:17:09.152262 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:09.152228    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerID="1645434bc2878c1f39584f2d69bc7985207bf2b6546f5f9cd38284177a9c0c82" exitCode=6
Jul 18 01:17:09.152705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:09.152310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" event={"ID":"2ffa64cd-b809-490a-9848-e78d0bc3052d","Type":"ContainerDied","Data":"1645434bc2878c1f39584f2d69bc7985207bf2b6546f5f9cd38284177a9c0c82"}
Jul 18 01:17:09.152705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:09.152368    2554 scope.go:117] "RemoveContainer" containerID="3fcc4ded7363ce0f4a4a2021968139c2aaefccbf674239ab072181cfca20c6bc"
Jul 18 01:17:09.152705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:09.152555    2554 scope.go:117] "RemoveContainer" containerID="1645434bc2878c1f39584f2d69bc7985207bf2b6546f5f9cd38284177a9c0c82"
Jul 18 01:17:09.152866 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:09.152831    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn_odh-ai-gateway-infra(2ffa64cd-b809-490a-9848-e78d0bc3052d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d"
Jul 18 01:17:10.159468 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:10.159433    2554 scope.go:117] "RemoveContainer" containerID="1645434bc2878c1f39584f2d69bc7985207bf2b6546f5f9cd38284177a9c0c82"
Jul 18 01:17:10.159896 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:10.159713    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn_odh-ai-gateway-infra(2ffa64cd-b809-490a-9848-e78d0bc3052d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d"
Jul 18 01:17:11.330351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:11.330310    2554 scope.go:117] "RemoveContainer" containerID="6640f5f1149f3858627adebc242f0bbe09425e0a0066a63d7cc8c1b5e4dbeb4b"
Jul 18 01:17:12.171522 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:12.171437    2554 generic.go:356] "Generic (PLEG): container finished" podID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerID="c07f88358e891d80b7482adf4702775b80a4e8eb6467bec5337849ac9c305722" exitCode=7
Jul 18 01:17:12.171522 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:12.171510    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" event={"ID":"8cb5332b-4e51-443e-9b61-e9bb35faa510","Type":"ContainerDied","Data":"c07f88358e891d80b7482adf4702775b80a4e8eb6467bec5337849ac9c305722"}
Jul 18 01:17:12.171759 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:12.171568    2554 scope.go:117] "RemoveContainer" containerID="6640f5f1149f3858627adebc242f0bbe09425e0a0066a63d7cc8c1b5e4dbeb4b"
Jul 18 01:17:12.171855 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:12.171839    2554 scope.go:117] "RemoveContainer" containerID="c07f88358e891d80b7482adf4702775b80a4e8eb6467bec5337849ac9c305722"
Jul 18 01:17:12.172103 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:12.172077    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-wbbsq_odh-ai-gateway-infra(8cb5332b-4e51-443e-9b61-e9bb35faa510)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510"
Jul 18 01:17:13.194889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:13.194854    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"]
Jul 18 01:17:13.336984 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:13.336954    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"
Jul 18 01:17:13.417366 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:13.417327    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5332b-4e51-443e-9b61-e9bb35faa510-maas-api-service-ca\") pod \"8cb5332b-4e51-443e-9b61-e9bb35faa510\" (UID: \"8cb5332b-4e51-443e-9b61-e9bb35faa510\") "
Jul 18 01:17:13.417722 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:13.417696    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5332b-4e51-443e-9b61-e9bb35faa510-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cb5332b-4e51-443e-9b61-e9bb35faa510" (UID: "8cb5332b-4e51-443e-9b61-e9bb35faa510"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:17:13.518241 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:13.518208    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5332b-4e51-443e-9b61-e9bb35faa510-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:17:14.183563 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:14.183529    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"
Jul 18 01:17:14.183766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:14.183563    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq" event={"ID":"8cb5332b-4e51-443e-9b61-e9bb35faa510","Type":"ContainerDied","Data":"c34f7ce1db3cd422b7fd75679c40fe2cde51e0e4735b469e9803454dd41e1bd8"}
Jul 18 01:17:14.183766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:14.183643    2554 scope.go:117] "RemoveContainer" containerID="c07f88358e891d80b7482adf4702775b80a4e8eb6467bec5337849ac9c305722"
Jul 18 01:17:14.207418 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:14.207389    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"]
Jul 18 01:17:14.212741 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:14.212718    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-wbbsq"]
Jul 18 01:17:14.336376 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:14.336346    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" path="/var/lib/kubelet/pods/8cb5332b-4e51-443e-9b61-e9bb35faa510/volumes"
Jul 18 01:17:22.330566 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:22.330537    2554 scope.go:117] "RemoveContainer" containerID="1645434bc2878c1f39584f2d69bc7985207bf2b6546f5f9cd38284177a9c0c82"
Jul 18 01:17:23.238924 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:23.238891    2554 generic.go:356] "Generic (PLEG): container finished" podID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerID="452f4ed212f98f9fe4d292632967723b1af119be72e9bc40c3a0ada9c82f7050" exitCode=6
Jul 18 01:17:23.239085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:23.238967    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" event={"ID":"2ffa64cd-b809-490a-9848-e78d0bc3052d","Type":"ContainerDied","Data":"452f4ed212f98f9fe4d292632967723b1af119be72e9bc40c3a0ada9c82f7050"}
Jul 18 01:17:23.239085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:23.239021    2554 scope.go:117] "RemoveContainer" containerID="1645434bc2878c1f39584f2d69bc7985207bf2b6546f5f9cd38284177a9c0c82"
Jul 18 01:17:23.239391 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:23.239371    2554 scope.go:117] "RemoveContainer" containerID="452f4ed212f98f9fe4d292632967723b1af119be72e9bc40c3a0ada9c82f7050"
Jul 18 01:17:23.239666 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:23.239610    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn_odh-ai-gateway-infra(2ffa64cd-b809-490a-9848-e78d0bc3052d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d"
Jul 18 01:17:24.263053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:24.263018    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"]
Jul 18 01:17:24.393437 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:24.393409    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"
Jul 18 01:17:24.526318 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:24.526242    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa64cd-b809-490a-9848-e78d0bc3052d-maas-api-service-ca\") pod \"2ffa64cd-b809-490a-9848-e78d0bc3052d\" (UID: \"2ffa64cd-b809-490a-9848-e78d0bc3052d\") "
Jul 18 01:17:24.526573 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:24.526552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffa64cd-b809-490a-9848-e78d0bc3052d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ffa64cd-b809-490a-9848-e78d0bc3052d" (UID: "2ffa64cd-b809-490a-9848-e78d0bc3052d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:17:24.627373 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:24.627342    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ffa64cd-b809-490a-9848-e78d0bc3052d-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:17:25.255367 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:25.255337    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"
Jul 18 01:17:25.255556 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:25.255373    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn" event={"ID":"2ffa64cd-b809-490a-9848-e78d0bc3052d","Type":"ContainerDied","Data":"b47db9c2bc03ed742455b59e98b97286b4427a3e4089c43d6661de01cf311441"}
Jul 18 01:17:25.255556 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:25.255433    2554 scope.go:117] "RemoveContainer" containerID="452f4ed212f98f9fe4d292632967723b1af119be72e9bc40c3a0ada9c82f7050"
Jul 18 01:17:25.280153 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:25.280122    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"]
Jul 18 01:17:25.285970 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:25.285947    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-cs9cn"]
Jul 18 01:17:26.336974 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:26.336946    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" path="/var/lib/kubelet/pods/2ffa64cd-b809-490a-9848-e78d0bc3052d/volumes"
Jul 18 01:17:28.637936 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.637898    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"]
Jul 18 01:17:28.638313 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638292    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:28.638313 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638303    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638317    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638323    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638332    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638338    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638344    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638349    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638356    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638361    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638373    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638378    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638385    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.638395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638390    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.639019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638454    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.639019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638462    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.639019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638469    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:28.639019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638479    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:28.639019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.638485    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2c2a2a-c3df-4d43-b637-59f6be3ff989" containerName="revoke-keys"
Jul 18 01:17:28.642889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.642871    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"
Jul 18 01:17:28.645864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.645837    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:17:28.647791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.647744    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"]
Jul 18 01:17:28.770850 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.770811    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bd393-d9b6-41dd-9b34-3e2cc2389d4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd\" (UID: \"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"
Jul 18 01:17:28.871735 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.871693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bd393-d9b6-41dd-9b34-3e2cc2389d4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd\" (UID: \"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"
Jul 18 01:17:28.872334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.872312    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bd393-d9b6-41dd-9b34-3e2cc2389d4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd\" (UID: \"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"
Jul 18 01:17:28.955007 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:28.954928    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"
Jul 18 01:17:29.291403 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:29.291376    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"]
Jul 18 01:17:29.293484 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:17:29.293456    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74bd393_d9b6_41dd_9b34_3e2cc2389d4b.slice/crio-e95f5e4f7a1ff98cfd660ae70a8e32735ace54ddc3a376454fd480ae20101df1 WatchSource:0}: Error finding container e95f5e4f7a1ff98cfd660ae70a8e32735ace54ddc3a376454fd480ae20101df1: Status 404 returned error can't find the container with id e95f5e4f7a1ff98cfd660ae70a8e32735ace54ddc3a376454fd480ae20101df1
Jul 18 01:17:30.282911 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:30.282877    2554 generic.go:356] "Generic (PLEG): container finished" podID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerID="ad41bad2e401e7af07a01c91f6c474ca541ba17032cd136bb3eca616791312e4" exitCode=6
Jul 18 01:17:30.285201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:30.282919    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" event={"ID":"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b","Type":"ContainerDied","Data":"ad41bad2e401e7af07a01c91f6c474ca541ba17032cd136bb3eca616791312e4"}
Jul 18 01:17:30.285201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:30.282975    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" event={"ID":"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b","Type":"ContainerStarted","Data":"e95f5e4f7a1ff98cfd660ae70a8e32735ace54ddc3a376454fd480ae20101df1"}
Jul 18 01:17:30.285201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:30.283172    2554 scope.go:117] "RemoveContainer" containerID="ad41bad2e401e7af07a01c91f6c474ca541ba17032cd136bb3eca616791312e4"
Jul 18 01:17:31.290079 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:31.289988    2554 generic.go:356] "Generic (PLEG): container finished" podID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerID="883f517ffec0bbb76c1c097a1b3ef8aed9ca0c8d94c9849ba40eb536e694c990" exitCode=6
Jul 18 01:17:31.290557 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:31.290072    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" event={"ID":"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b","Type":"ContainerDied","Data":"883f517ffec0bbb76c1c097a1b3ef8aed9ca0c8d94c9849ba40eb536e694c990"}
Jul 18 01:17:31.290557 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:31.290139    2554 scope.go:117] "RemoveContainer" containerID="ad41bad2e401e7af07a01c91f6c474ca541ba17032cd136bb3eca616791312e4"
Jul 18 01:17:31.290557 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:31.290329    2554 scope.go:117] "RemoveContainer" containerID="883f517ffec0bbb76c1c097a1b3ef8aed9ca0c8d94c9849ba40eb536e694c990"
Jul 18 01:17:31.290752 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:31.290603    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd_odh-ai-gateway-infra(c74bd393-d9b6-41dd-9b34-3e2cc2389d4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b"
Jul 18 01:17:32.297036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:32.297007    2554 scope.go:117] "RemoveContainer" containerID="883f517ffec0bbb76c1c097a1b3ef8aed9ca0c8d94c9849ba40eb536e694c990"
Jul 18 01:17:32.297431 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:32.297290    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd_odh-ai-gateway-infra(c74bd393-d9b6-41dd-9b34-3e2cc2389d4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b"
Jul 18 01:17:44.700781 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.700749    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"]
Jul 18 01:17:44.701215 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.701202    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb5332b-4e51-443e-9b61-e9bb35faa510" containerName="revoke-keys"
Jul 18 01:17:44.701258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.701221    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffa64cd-b809-490a-9848-e78d0bc3052d" containerName="revoke-keys"
Jul 18 01:17:44.705727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.705711    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"
Jul 18 01:17:44.710458 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.710431    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"]
Jul 18 01:17:44.734007 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.733971    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf6616a-a223-4ec9-bdfe-d6f0990cb63c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gvrv2\" (UID: \"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"
Jul 18 01:17:44.835511 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.835415    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf6616a-a223-4ec9-bdfe-d6f0990cb63c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gvrv2\" (UID: \"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"
Jul 18 01:17:44.836060 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:44.836039    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf6616a-a223-4ec9-bdfe-d6f0990cb63c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gvrv2\" (UID: \"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"
Jul 18 01:17:45.017915 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:45.017880    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"
Jul 18 01:17:45.145926 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:45.145890    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"]
Jul 18 01:17:45.148694 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:17:45.148663    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf6616a_a223_4ec9_bdfe_d6f0990cb63c.slice/crio-5906f703db36f7fea7f0dbecd81194e4ea2cfff5f1a33f41b5d28b205bfdb6a2 WatchSource:0}: Error finding container 5906f703db36f7fea7f0dbecd81194e4ea2cfff5f1a33f41b5d28b205bfdb6a2: Status 404 returned error can't find the container with id 5906f703db36f7fea7f0dbecd81194e4ea2cfff5f1a33f41b5d28b205bfdb6a2
Jul 18 01:17:45.330893 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:45.330870    2554 scope.go:117] "RemoveContainer" containerID="883f517ffec0bbb76c1c097a1b3ef8aed9ca0c8d94c9849ba40eb536e694c990"
Jul 18 01:17:45.363962 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:45.363932    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" event={"ID":"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c","Type":"ContainerStarted","Data":"c864af346631288caad69b9c63e89584833fd8bc4c139acd9dfa9364d0b87289"}
Jul 18 01:17:45.364083 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:45.363986    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" event={"ID":"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c","Type":"ContainerStarted","Data":"5906f703db36f7fea7f0dbecd81194e4ea2cfff5f1a33f41b5d28b205bfdb6a2"}
Jul 18 01:17:45.379052 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:45.379010    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" podStartSLOduration=1.378994577 podStartE2EDuration="1.378994577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:17:45.377175498 +0000 UTC m=+2221.503110470" watchObservedRunningTime="2026-07-18 01:17:45.378994577 +0000 UTC m=+2221.504929556"
Jul 18 01:17:46.370175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:46.370140    2554 generic.go:356] "Generic (PLEG): container finished" podID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerID="831744d46a57becb6d7246176ae2c28694ffc05746d052f80390e5fd12cb6981" exitCode=6
Jul 18 01:17:46.370636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:46.370208    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" event={"ID":"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b","Type":"ContainerDied","Data":"831744d46a57becb6d7246176ae2c28694ffc05746d052f80390e5fd12cb6981"}
Jul 18 01:17:46.370636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:46.370267    2554 scope.go:117] "RemoveContainer" containerID="883f517ffec0bbb76c1c097a1b3ef8aed9ca0c8d94c9849ba40eb536e694c990"
Jul 18 01:17:46.370636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:46.370580    2554 scope.go:117] "RemoveContainer" containerID="831744d46a57becb6d7246176ae2c28694ffc05746d052f80390e5fd12cb6981"
Jul 18 01:17:46.370867 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:46.370844    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd_odh-ai-gateway-infra(c74bd393-d9b6-41dd-9b34-3e2cc2389d4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b"
Jul 18 01:17:46.372042 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:46.372016    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerID="c864af346631288caad69b9c63e89584833fd8bc4c139acd9dfa9364d0b87289" exitCode=7
Jul 18 01:17:46.372181 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:46.372052    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" event={"ID":"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c","Type":"ContainerDied","Data":"c864af346631288caad69b9c63e89584833fd8bc4c139acd9dfa9364d0b87289"}
Jul 18 01:17:46.372239 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:46.372221    2554 scope.go:117] "RemoveContainer" containerID="c864af346631288caad69b9c63e89584833fd8bc4c139acd9dfa9364d0b87289"
Jul 18 01:17:47.381669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.381551    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerID="b18028ed8499a4975e15f39ef9a21036dababec24d13b8348e69827dea8be3e4" exitCode=7
Jul 18 01:17:47.381669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.381590    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" event={"ID":"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c","Type":"ContainerDied","Data":"b18028ed8499a4975e15f39ef9a21036dababec24d13b8348e69827dea8be3e4"}
Jul 18 01:17:47.381669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.381651    2554 scope.go:117] "RemoveContainer" containerID="c864af346631288caad69b9c63e89584833fd8bc4c139acd9dfa9364d0b87289"
Jul 18 01:17:47.382246 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.381890    2554 scope.go:117] "RemoveContainer" containerID="b18028ed8499a4975e15f39ef9a21036dababec24d13b8348e69827dea8be3e4"
Jul 18 01:17:47.382246 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:47.382107    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gvrv2_odh-ai-gateway-infra(ecf6616a-a223-4ec9-bdfe-d6f0990cb63c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c"
Jul 18 01:17:47.395796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.395767    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"]
Jul 18 01:17:47.537182 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.537157    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"
Jul 18 01:17:47.563726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.563692    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bd393-d9b6-41dd-9b34-3e2cc2389d4b-maas-api-service-ca\") pod \"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b\" (UID: \"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b\") "
Jul 18 01:17:47.564074 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.564049    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74bd393-d9b6-41dd-9b34-3e2cc2389d4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" (UID: "c74bd393-d9b6-41dd-9b34-3e2cc2389d4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:17:47.564187 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:47.564170    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bd393-d9b6-41dd-9b34-3e2cc2389d4b-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:17:48.389024 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:48.389001    2554 scope.go:117] "RemoveContainer" containerID="b18028ed8499a4975e15f39ef9a21036dababec24d13b8348e69827dea8be3e4"
Jul 18 01:17:48.389406 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:48.389198    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gvrv2_odh-ai-gateway-infra(ecf6616a-a223-4ec9-bdfe-d6f0990cb63c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c"
Jul 18 01:17:48.390056 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:48.390036    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"
Jul 18 01:17:48.390185 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:48.390061    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd" event={"ID":"c74bd393-d9b6-41dd-9b34-3e2cc2389d4b","Type":"ContainerDied","Data":"e95f5e4f7a1ff98cfd660ae70a8e32735ace54ddc3a376454fd480ae20101df1"}
Jul 18 01:17:48.390185 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:48.390121    2554 scope.go:117] "RemoveContainer" containerID="831744d46a57becb6d7246176ae2c28694ffc05746d052f80390e5fd12cb6981"
Jul 18 01:17:48.420873 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:48.420823    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"]
Jul 18 01:17:48.422697 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:48.422674    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9mnd"]
Jul 18 01:17:50.336989 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:50.336954    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" path="/var/lib/kubelet/pods/c74bd393-d9b6-41dd-9b34-3e2cc2389d4b/volumes"
Jul 18 01:17:56.688387 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.688347    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"]
Jul 18 01:17:56.688793 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.688763    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:17:56.688793 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.688775    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:17:56.688793 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.688793    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:17:56.688901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.688799    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:17:56.688901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.688864    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:17:56.688901 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.688872    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:17:56.691907 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.691890    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"
Jul 18 01:17:56.697921 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.697891    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"]
Jul 18 01:17:56.761308 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.761268    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a0f8f-af86-4484-8d2b-058c62886950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk\" (UID: \"9a8a0f8f-af86-4484-8d2b-058c62886950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"
Jul 18 01:17:56.862148 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.862110    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a0f8f-af86-4484-8d2b-058c62886950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk\" (UID: \"9a8a0f8f-af86-4484-8d2b-058c62886950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"
Jul 18 01:17:56.862804 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:56.862785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a0f8f-af86-4484-8d2b-058c62886950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk\" (UID: \"9a8a0f8f-af86-4484-8d2b-058c62886950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"
Jul 18 01:17:57.005249 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:57.005214    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"
Jul 18 01:17:57.154069 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:57.153973    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"]
Jul 18 01:17:57.154196 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:17:57.154166    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8a0f8f_af86_4484_8d2b_058c62886950.slice/crio-91f1323b12464bf0005aca520cd5b704cb8128ecde0fb09bae62832a51068f67 WatchSource:0}: Error finding container 91f1323b12464bf0005aca520cd5b704cb8128ecde0fb09bae62832a51068f67: Status 404 returned error can't find the container with id 91f1323b12464bf0005aca520cd5b704cb8128ecde0fb09bae62832a51068f67
Jul 18 01:17:57.435265 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:57.435229    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" event={"ID":"9a8a0f8f-af86-4484-8d2b-058c62886950","Type":"ContainerStarted","Data":"12705eed5af2b56b5e3dfab1f279bbd0a5670598da4f385ff7049949c73e16e5"}
Jul 18 01:17:57.435436 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:57.435281    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" event={"ID":"9a8a0f8f-af86-4484-8d2b-058c62886950","Type":"ContainerStarted","Data":"91f1323b12464bf0005aca520cd5b704cb8128ecde0fb09bae62832a51068f67"}
Jul 18 01:17:57.450567 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:57.450521    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" podStartSLOduration=1.4505057940000001 podStartE2EDuration="1.450505794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:17:57.448472563 +0000 UTC m=+2233.574407544" watchObservedRunningTime="2026-07-18 01:17:57.450505794 +0000 UTC m=+2233.576440773"
Jul 18 01:17:58.441597 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:58.441562    2554 generic.go:356] "Generic (PLEG): container finished" podID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerID="12705eed5af2b56b5e3dfab1f279bbd0a5670598da4f385ff7049949c73e16e5" exitCode=6
Jul 18 01:17:58.442000 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:58.441650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" event={"ID":"9a8a0f8f-af86-4484-8d2b-058c62886950","Type":"ContainerDied","Data":"12705eed5af2b56b5e3dfab1f279bbd0a5670598da4f385ff7049949c73e16e5"}
Jul 18 01:17:58.442000 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:58.441816    2554 scope.go:117] "RemoveContainer" containerID="12705eed5af2b56b5e3dfab1f279bbd0a5670598da4f385ff7049949c73e16e5"
Jul 18 01:17:59.449782 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:59.449695    2554 generic.go:356] "Generic (PLEG): container finished" podID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerID="39f4d5340edc368c0baad453c67b87fe8a448d812f1ed6436b45dc5744478982" exitCode=6
Jul 18 01:17:59.449782 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:59.449757    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" event={"ID":"9a8a0f8f-af86-4484-8d2b-058c62886950","Type":"ContainerDied","Data":"39f4d5340edc368c0baad453c67b87fe8a448d812f1ed6436b45dc5744478982"}
Jul 18 01:17:59.450268 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:59.449843    2554 scope.go:117] "RemoveContainer" containerID="12705eed5af2b56b5e3dfab1f279bbd0a5670598da4f385ff7049949c73e16e5"
Jul 18 01:17:59.450268 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:17:59.450045    2554 scope.go:117] "RemoveContainer" containerID="39f4d5340edc368c0baad453c67b87fe8a448d812f1ed6436b45dc5744478982"
Jul 18 01:17:59.450378 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:17:59.450297    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk_odh-ai-gateway-infra(9a8a0f8f-af86-4484-8d2b-058c62886950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950"
Jul 18 01:18:00.455973 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:00.455949    2554 scope.go:117] "RemoveContainer" containerID="39f4d5340edc368c0baad453c67b87fe8a448d812f1ed6436b45dc5744478982"
Jul 18 01:18:00.456356 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:00.456168    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk_odh-ai-gateway-infra(9a8a0f8f-af86-4484-8d2b-058c62886950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950"
Jul 18 01:18:03.330798 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:03.330767    2554 scope.go:117] "RemoveContainer" containerID="b18028ed8499a4975e15f39ef9a21036dababec24d13b8348e69827dea8be3e4"
Jul 18 01:18:04.478649 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:04.478593    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerID="d394533a54d19e8cbf92a8a83774c36b6f56ea759c63c08aa5aed36e1af6459f" exitCode=7
Jul 18 01:18:04.479115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:04.478651    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" event={"ID":"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c","Type":"ContainerDied","Data":"d394533a54d19e8cbf92a8a83774c36b6f56ea759c63c08aa5aed36e1af6459f"}
Jul 18 01:18:04.479115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:04.478712    2554 scope.go:117] "RemoveContainer" containerID="b18028ed8499a4975e15f39ef9a21036dababec24d13b8348e69827dea8be3e4"
Jul 18 01:18:04.479115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:04.479034    2554 scope.go:117] "RemoveContainer" containerID="d394533a54d19e8cbf92a8a83774c36b6f56ea759c63c08aa5aed36e1af6459f"
Jul 18 01:18:04.479347 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:04.479323    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gvrv2_odh-ai-gateway-infra(ecf6616a-a223-4ec9-bdfe-d6f0990cb63c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c"
Jul 18 01:18:05.502162 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:05.502128    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"]
Jul 18 01:18:05.667225 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:05.667199    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"
Jul 18 01:18:05.754813 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:05.754736    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf6616a-a223-4ec9-bdfe-d6f0990cb63c-maas-api-service-ca\") pod \"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c\" (UID: \"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c\") "
Jul 18 01:18:05.755043 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:05.755022    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf6616a-a223-4ec9-bdfe-d6f0990cb63c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" (UID: "ecf6616a-a223-4ec9-bdfe-d6f0990cb63c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:18:05.856144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:05.856106    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf6616a-a223-4ec9-bdfe-d6f0990cb63c-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:06.491049 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:06.491024    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"
Jul 18 01:18:06.491211 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:06.491057    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2" event={"ID":"ecf6616a-a223-4ec9-bdfe-d6f0990cb63c","Type":"ContainerDied","Data":"5906f703db36f7fea7f0dbecd81194e4ea2cfff5f1a33f41b5d28b205bfdb6a2"}
Jul 18 01:18:06.491211 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:06.491113    2554 scope.go:117] "RemoveContainer" containerID="d394533a54d19e8cbf92a8a83774c36b6f56ea759c63c08aa5aed36e1af6459f"
Jul 18 01:18:06.507818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:06.507793    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"]
Jul 18 01:18:06.512636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:06.512593    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gvrv2"]
Jul 18 01:18:08.342329 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:08.342291    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" path="/var/lib/kubelet/pods/ecf6616a-a223-4ec9-bdfe-d6f0990cb63c/volumes"
Jul 18 01:18:15.330723 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:15.330688    2554 scope.go:117] "RemoveContainer" containerID="39f4d5340edc368c0baad453c67b87fe8a448d812f1ed6436b45dc5744478982"
Jul 18 01:18:16.542681 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:16.542650    2554 generic.go:356] "Generic (PLEG): container finished" podID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerID="1b6a8ac1aef85918495634e35d848448dbd3365bbaa07e0e6dd0abfefef1a947" exitCode=6
Jul 18 01:18:16.543106 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:16.542711    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" event={"ID":"9a8a0f8f-af86-4484-8d2b-058c62886950","Type":"ContainerDied","Data":"1b6a8ac1aef85918495634e35d848448dbd3365bbaa07e0e6dd0abfefef1a947"}
Jul 18 01:18:16.543106 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:16.542765    2554 scope.go:117] "RemoveContainer" containerID="39f4d5340edc368c0baad453c67b87fe8a448d812f1ed6436b45dc5744478982"
Jul 18 01:18:16.543106 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:16.543073    2554 scope.go:117] "RemoveContainer" containerID="1b6a8ac1aef85918495634e35d848448dbd3365bbaa07e0e6dd0abfefef1a947"
Jul 18 01:18:16.543331 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:16.543306    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk_odh-ai-gateway-infra(9a8a0f8f-af86-4484-8d2b-058c62886950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950"
Jul 18 01:18:17.571491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:17.571446    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"]
Jul 18 01:18:17.708860 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:17.708834    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"
Jul 18 01:18:17.770769 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:17.770737    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a0f8f-af86-4484-8d2b-058c62886950-maas-api-service-ca\") pod \"9a8a0f8f-af86-4484-8d2b-058c62886950\" (UID: \"9a8a0f8f-af86-4484-8d2b-058c62886950\") "
Jul 18 01:18:17.771092 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:17.771069    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8a0f8f-af86-4484-8d2b-058c62886950-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a8a0f8f-af86-4484-8d2b-058c62886950" (UID: "9a8a0f8f-af86-4484-8d2b-058c62886950"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:18:17.871744 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:17.871661    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a0f8f-af86-4484-8d2b-058c62886950-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:18.554990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.554961    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"
Jul 18 01:18:18.554990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.554977    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk" event={"ID":"9a8a0f8f-af86-4484-8d2b-058c62886950","Type":"ContainerDied","Data":"91f1323b12464bf0005aca520cd5b704cb8128ecde0fb09bae62832a51068f67"}
Jul 18 01:18:18.555500 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.555032    2554 scope.go:117] "RemoveContainer" containerID="1b6a8ac1aef85918495634e35d848448dbd3365bbaa07e0e6dd0abfefef1a947"
Jul 18 01:18:18.576218 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.576196    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"]
Jul 18 01:18:18.580581 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.580558    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d6phk"]
Jul 18 01:18:18.674746 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.674670    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"]
Jul 18 01:18:18.675085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675072    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:18.675128 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675087    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:18.675128 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675095    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:18.675128 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675101    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:18.675128 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675108    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:18.675128 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675114    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675140    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675146    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675153    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675158    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675209    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675217    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675225    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675233    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74bd393-d9b6-41dd-9b34-3e2cc2389d4b" containerName="revoke-keys"
Jul 18 01:18:18.675282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.675239    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:18.679293 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.679276    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"
Jul 18 01:18:18.681990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.681965    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:18:18.684446 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.684421    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"]
Jul 18 01:18:18.780970 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.780936    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf8f21-c42d-47f2-af06-31c200a020a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-w9865\" (UID: \"d6cf8f21-c42d-47f2-af06-31c200a020a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"
Jul 18 01:18:18.882102 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.882065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf8f21-c42d-47f2-af06-31c200a020a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-w9865\" (UID: \"d6cf8f21-c42d-47f2-af06-31c200a020a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"
Jul 18 01:18:18.882683 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.882665    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf8f21-c42d-47f2-af06-31c200a020a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-w9865\" (UID: \"d6cf8f21-c42d-47f2-af06-31c200a020a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"
Jul 18 01:18:18.990677 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:18.990649    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"
Jul 18 01:18:19.119488 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:19.119455    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"]
Jul 18 01:18:19.121131 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:18:19.121092    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cf8f21_c42d_47f2_af06_31c200a020a5.slice/crio-997c20ecbb3b932b6732c0d3be08f915d586f116a13b9cdc702eba27e5089300 WatchSource:0}: Error finding container 997c20ecbb3b932b6732c0d3be08f915d586f116a13b9cdc702eba27e5089300: Status 404 returned error can't find the container with id 997c20ecbb3b932b6732c0d3be08f915d586f116a13b9cdc702eba27e5089300
Jul 18 01:18:19.561364 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:19.561323    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" event={"ID":"d6cf8f21-c42d-47f2-af06-31c200a020a5","Type":"ContainerStarted","Data":"ee38050b6023fa3ccb4ef6dc0405248c36d4590b825c701b6334bead762af85f"}
Jul 18 01:18:19.561560 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:19.561388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" event={"ID":"d6cf8f21-c42d-47f2-af06-31c200a020a5","Type":"ContainerStarted","Data":"997c20ecbb3b932b6732c0d3be08f915d586f116a13b9cdc702eba27e5089300"}
Jul 18 01:18:19.581016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:19.580899    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" podStartSLOduration=1.580870006 podStartE2EDuration="1.580870006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:18:19.580815746 +0000 UTC m=+2255.706750735" watchObservedRunningTime="2026-07-18 01:18:19.580870006 +0000 UTC m=+2255.706804981"
Jul 18 01:18:20.336447 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:20.336415    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" path="/var/lib/kubelet/pods/9a8a0f8f-af86-4484-8d2b-058c62886950/volumes"
Jul 18 01:18:20.568913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:20.568879    2554 generic.go:356] "Generic (PLEG): container finished" podID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerID="ee38050b6023fa3ccb4ef6dc0405248c36d4590b825c701b6334bead762af85f" exitCode=6
Jul 18 01:18:20.569093 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:20.568964    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" event={"ID":"d6cf8f21-c42d-47f2-af06-31c200a020a5","Type":"ContainerDied","Data":"ee38050b6023fa3ccb4ef6dc0405248c36d4590b825c701b6334bead762af85f"}
Jul 18 01:18:20.569142 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:20.569121    2554 scope.go:117] "RemoveContainer" containerID="ee38050b6023fa3ccb4ef6dc0405248c36d4590b825c701b6334bead762af85f"
Jul 18 01:18:21.575803 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:21.575771    2554 generic.go:356] "Generic (PLEG): container finished" podID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerID="ae2ec7c3e27d18da6a8923d63c6dad06d3cdb62362e5e246b09f6b7bf05f85f8" exitCode=6
Jul 18 01:18:21.576249 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:21.575845    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" event={"ID":"d6cf8f21-c42d-47f2-af06-31c200a020a5","Type":"ContainerDied","Data":"ae2ec7c3e27d18da6a8923d63c6dad06d3cdb62362e5e246b09f6b7bf05f85f8"}
Jul 18 01:18:21.576249 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:21.575901    2554 scope.go:117] "RemoveContainer" containerID="ee38050b6023fa3ccb4ef6dc0405248c36d4590b825c701b6334bead762af85f"
Jul 18 01:18:21.576249 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:21.576097    2554 scope.go:117] "RemoveContainer" containerID="ae2ec7c3e27d18da6a8923d63c6dad06d3cdb62362e5e246b09f6b7bf05f85f8"
Jul 18 01:18:21.576393 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:21.576334    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-w9865_odh-ai-gateway-infra(d6cf8f21-c42d-47f2-af06-31c200a020a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5"
Jul 18 01:18:22.582673 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:22.582646    2554 scope.go:117] "RemoveContainer" containerID="ae2ec7c3e27d18da6a8923d63c6dad06d3cdb62362e5e246b09f6b7bf05f85f8"
Jul 18 01:18:22.583057 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:22.582907    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-w9865_odh-ai-gateway-infra(d6cf8f21-c42d-47f2-af06-31c200a020a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5"
Jul 18 01:18:24.668957 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:24.668920    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"]
Jul 18 01:18:24.669423 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:24.669130    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy" containerID="cri-o://6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268" gracePeriod=30
Jul 18 01:18:29.920016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.919719    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:18:29.990187 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990159    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-socket\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990220    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-envoy\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990248    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1522746-3999-46cf-8cb9-1704658f2992-istiod-ca-cert\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990269    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-data\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1522746-3999-46cf-8cb9-1704658f2992-istio-podinfo\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990318    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-istio-token\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990599 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990357    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-credential-socket\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990599 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990426    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-certs\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990599 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990468    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxbbv\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-kube-api-access-gxbbv\") pod \"a1522746-3999-46cf-8cb9-1704658f2992\" (UID: \"a1522746-3999-46cf-8cb9-1704658f2992\") "
Jul 18 01:18:29.990790 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990719    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:18:29.990845 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990815    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:18:29.990903 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990860    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1522746-3999-46cf-8cb9-1704658f2992-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:18:29.990956 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990901    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:29.990956 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.990921    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:29.991230 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.991200    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:18:29.991348 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.991296    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:18:29.992897 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.992873    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:18:29.993107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.993081    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-kube-api-access-gxbbv" (OuterVolumeSpecName: "kube-api-access-gxbbv") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "kube-api-access-gxbbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:18:29.993107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.993098    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1522746-3999-46cf-8cb9-1704658f2992-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:18:29.993253 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:29.993118    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1522746-3999-46cf-8cb9-1704658f2992" (UID: "a1522746-3999-46cf-8cb9-1704658f2992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:18:30.092382 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.092348    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.092382 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.092376    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1522746-3999-46cf-8cb9-1704658f2992-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.092591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.092400    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.092591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.092410    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1522746-3999-46cf-8cb9-1704658f2992-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.092591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.092418    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.092591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.092427    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1522746-3999-46cf-8cb9-1704658f2992-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.092591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.092435    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxbbv\" (UniqueName: \"kubernetes.io/projected/a1522746-3999-46cf-8cb9-1704658f2992-kube-api-access-gxbbv\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.625554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.625470    2554 generic.go:356] "Generic (PLEG): container finished" podID="a1522746-3999-46cf-8cb9-1704658f2992" containerID="6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268" exitCode=0
Jul 18 01:18:30.625554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.625517    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"
Jul 18 01:18:30.625766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.625551    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" event={"ID":"a1522746-3999-46cf-8cb9-1704658f2992","Type":"ContainerDied","Data":"6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268"}
Jul 18 01:18:30.625766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.625646    2554 scope.go:117] "RemoveContainer" containerID="6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268"
Jul 18 01:18:30.625766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.625610    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465" event={"ID":"a1522746-3999-46cf-8cb9-1704658f2992","Type":"ContainerDied","Data":"8911a2f7f4e102897c5fffeb3beb5d0acfee4be510e78404a2629e0da8c15971"}
Jul 18 01:18:30.641118 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.641094    2554 scope.go:117] "RemoveContainer" containerID="6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9"
Jul 18 01:18:30.644473 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.644451    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"]
Jul 18 01:18:30.648160 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.648130    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4701d191-openshift-default-6b94dc8697-sb465"]
Jul 18 01:18:30.650296 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.650281    2554 scope.go:117] "RemoveContainer" containerID="6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268"
Jul 18 01:18:30.650577 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:30.650558    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268\": container with ID starting with 6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268 not found: ID does not exist" containerID="6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268"
Jul 18 01:18:30.650660 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.650591    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268"} err="failed to get container status \"6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268\": rpc error: code = NotFound desc = could not find container \"6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268\": container with ID starting with 6f852f934f666820f9a8abb5b8a8ff30fce80f2c9220522e4570a1f104c11268 not found: ID does not exist"
Jul 18 01:18:30.650660 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.650611    2554 scope.go:117] "RemoveContainer" containerID="6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9"
Jul 18 01:18:30.650883 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:30.650861    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9\": container with ID starting with 6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9 not found: ID does not exist" containerID="6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9"
Jul 18 01:18:30.650922 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:30.650889    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9"} err="failed to get container status \"6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9\": rpc error: code = NotFound desc = could not find container \"6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9\": container with ID starting with 6a56e51b4735c6d537863a07464bb16f4771dc700b2b12610412211f874233f9 not found: ID does not exist"
Jul 18 01:18:32.337453 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:32.337409    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1522746-3999-46cf-8cb9-1704658f2992" path="/var/lib/kubelet/pods/a1522746-3999-46cf-8cb9-1704658f2992/volumes"
Jul 18 01:18:36.210739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.210704    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"]
Jul 18 01:18:36.211360 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211342    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy"
Jul 18 01:18:36.211406 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211365    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy"
Jul 18 01:18:36.211461 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211412    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:36.211503 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211463    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:36.211503 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211482    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:36.211503 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211492    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:36.211643 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211630    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy"
Jul 18 01:18:36.211678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211646    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy"
Jul 18 01:18:36.211678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211662    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8a0f8f-af86-4484-8d2b-058c62886950" containerName="revoke-keys"
Jul 18 01:18:36.211678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.211673    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf6616a-a223-4ec9-bdfe-d6f0990cb63c" containerName="revoke-keys"
Jul 18 01:18:36.216652 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.216634    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.219882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.219862    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4e26d7cc-openshift-default-dockercfg-p7qlp\""
Jul 18 01:18:36.226282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.226256    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"]
Jul 18 01:18:36.354971 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.354939    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9st\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-kube-api-access-tq9st\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355002    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-credential-socket\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355031    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-token\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355062    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-certs\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355078    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istiod-ca-cert\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355096    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-data\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355316 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355180    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-socket\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355316 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355214    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-podinfo\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.355316 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.355256    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-envoy\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.455855 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.455823    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-credential-socket\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.455855 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.455858    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-token\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.455878    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-certs\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.455895    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istiod-ca-cert\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.455920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-data\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456004    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-socket\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456036    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-podinfo\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456339 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456084    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-envoy\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456339 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456116    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9st\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-kube-api-access-tq9st\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456339 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456261    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-credential-socket\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456339 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456294    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-certs\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456354    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-data\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456696 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456672    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istiod-ca-cert\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.456778 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.456721    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-socket\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.458230 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.458211    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-envoy\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.458468 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.458451    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-podinfo\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.463641 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.463574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-token\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.463724 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.463706    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9st\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-kube-api-access-tq9st\") pod \"e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") " pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.528893 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.528865    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:36.666733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.666711    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"]
Jul 18 01:18:36.668932 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:18:36.668905    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94bc8b3_2e7b_4da1_bd09_df10eb1b2868.slice/crio-9ba92699fab6e03ee9804c0319a720365af17568174b4428541c65f4463e0ea5 WatchSource:0}: Error finding container 9ba92699fab6e03ee9804c0319a720365af17568174b4428541c65f4463e0ea5: Status 404 returned error can't find the container with id 9ba92699fab6e03ee9804c0319a720365af17568174b4428541c65f4463e0ea5
Jul 18 01:18:36.671458 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.671424    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:18:36.671569 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.671497    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:18:36.671569 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:36.671526    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:18:37.330245 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:37.330220    2554 scope.go:117] "RemoveContainer" containerID="ae2ec7c3e27d18da6a8923d63c6dad06d3cdb62362e5e246b09f6b7bf05f85f8"
Jul 18 01:18:37.663102 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:37.663057    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" event={"ID":"d6cf8f21-c42d-47f2-af06-31c200a020a5","Type":"ContainerStarted","Data":"d66460eee1245562845c21fab851733c08110fb3f22231bfe3b2ac45f0f65fa2"}
Jul 18 01:18:37.664678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:37.664645    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" event={"ID":"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868","Type":"ContainerStarted","Data":"93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12"}
Jul 18 01:18:37.664841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:37.664702    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" event={"ID":"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868","Type":"ContainerStarted","Data":"9ba92699fab6e03ee9804c0319a720365af17568174b4428541c65f4463e0ea5"}
Jul 18 01:18:37.742029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:37.741974    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podStartSLOduration=1.741956279 podStartE2EDuration="1.741956279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:18:37.7395323 +0000 UTC m=+2273.865467280" watchObservedRunningTime="2026-07-18 01:18:37.741956279 +0000 UTC m=+2273.867891257"
Jul 18 01:18:38.529439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.529407    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:38.534123 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.534098    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:38.671024 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.670993    2554 generic.go:356] "Generic (PLEG): container finished" podID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerID="d66460eee1245562845c21fab851733c08110fb3f22231bfe3b2ac45f0f65fa2" exitCode=6
Jul 18 01:18:38.671174 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.671064    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" event={"ID":"d6cf8f21-c42d-47f2-af06-31c200a020a5","Type":"ContainerDied","Data":"d66460eee1245562845c21fab851733c08110fb3f22231bfe3b2ac45f0f65fa2"}
Jul 18 01:18:38.671174 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.671123    2554 scope.go:117] "RemoveContainer" containerID="ae2ec7c3e27d18da6a8923d63c6dad06d3cdb62362e5e246b09f6b7bf05f85f8"
Jul 18 01:18:38.671321 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.671287    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:38.671586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.671562    2554 scope.go:117] "RemoveContainer" containerID="d66460eee1245562845c21fab851733c08110fb3f22231bfe3b2ac45f0f65fa2"
Jul 18 01:18:38.671848 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:38.671813    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-w9865_odh-ai-gateway-infra(d6cf8f21-c42d-47f2-af06-31c200a020a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5"
Jul 18 01:18:38.672674 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.672456    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:38.710644 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:38.710595    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"]
Jul 18 01:18:39.812840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:39.812817    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"
Jul 18 01:18:39.991793 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:39.991759    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf8f21-c42d-47f2-af06-31c200a020a5-maas-api-service-ca\") pod \"d6cf8f21-c42d-47f2-af06-31c200a020a5\" (UID: \"d6cf8f21-c42d-47f2-af06-31c200a020a5\") "
Jul 18 01:18:39.992111 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:39.992086    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cf8f21-c42d-47f2-af06-31c200a020a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6cf8f21-c42d-47f2-af06-31c200a020a5" (UID: "d6cf8f21-c42d-47f2-af06-31c200a020a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:18:40.093041 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:40.093003    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6cf8f21-c42d-47f2-af06-31c200a020a5-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:18:40.684950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:40.684845    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865" event={"ID":"d6cf8f21-c42d-47f2-af06-31c200a020a5","Type":"ContainerDied","Data":"997c20ecbb3b932b6732c0d3be08f915d586f116a13b9cdc702eba27e5089300"}
Jul 18 01:18:40.684950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:40.684932    2554 scope.go:117] "RemoveContainer" containerID="d66460eee1245562845c21fab851733c08110fb3f22231bfe3b2ac45f0f65fa2"
Jul 18 01:18:40.685183 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:40.684871    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"
Jul 18 01:18:40.700342 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:40.700319    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"]
Jul 18 01:18:40.702163 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:40.702144    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-w9865"]
Jul 18 01:18:41.825760 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.825729    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"]
Jul 18 01:18:41.826146 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826123    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:41.826146 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826136    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:41.826228 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826154    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy"
Jul 18 01:18:41.826228 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826160    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1522746-3999-46cf-8cb9-1704658f2992" containerName="istio-proxy"
Jul 18 01:18:41.826228 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826179    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:41.826228 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826185    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:41.826373 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826241    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:41.826373 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.826251    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:41.830516 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.830495    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"
Jul 18 01:18:41.833366 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.833345    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:18:41.841419 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:41.841396    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"]
Jul 18 01:18:42.013669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.013602    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2877c4fc-000f-45fd-b3ff-925e407fcb3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth\" (UID: \"2877c4fc-000f-45fd-b3ff-925e407fcb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"
Jul 18 01:18:42.114532 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.114452    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2877c4fc-000f-45fd-b3ff-925e407fcb3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth\" (UID: \"2877c4fc-000f-45fd-b3ff-925e407fcb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"
Jul 18 01:18:42.115092 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.115073    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2877c4fc-000f-45fd-b3ff-925e407fcb3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth\" (UID: \"2877c4fc-000f-45fd-b3ff-925e407fcb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"
Jul 18 01:18:42.141580 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.141554    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"
Jul 18 01:18:42.286366 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.286334    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"]
Jul 18 01:18:42.288140 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:18:42.288100    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2877c4fc_000f_45fd_b3ff_925e407fcb3f.slice/crio-4b912aa99e4e933f05aaa7daab56dab88b472f3f734cc49808d28706b0277cc6 WatchSource:0}: Error finding container 4b912aa99e4e933f05aaa7daab56dab88b472f3f734cc49808d28706b0277cc6: Status 404 returned error can't find the container with id 4b912aa99e4e933f05aaa7daab56dab88b472f3f734cc49808d28706b0277cc6
Jul 18 01:18:42.336267 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.336234    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" path="/var/lib/kubelet/pods/d6cf8f21-c42d-47f2-af06-31c200a020a5/volumes"
Jul 18 01:18:42.696704 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.696601    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" event={"ID":"2877c4fc-000f-45fd-b3ff-925e407fcb3f","Type":"ContainerStarted","Data":"5869fde8581cead83c30c5b191c8474c9c1275ad56e2b029f81d0ec7b07d4191"}
Jul 18 01:18:42.696704 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.696669    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" event={"ID":"2877c4fc-000f-45fd-b3ff-925e407fcb3f","Type":"ContainerStarted","Data":"4b912aa99e4e933f05aaa7daab56dab88b472f3f734cc49808d28706b0277cc6"}
Jul 18 01:18:42.713813 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:42.713770    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" podStartSLOduration=1.713752334 podStartE2EDuration="1.713752334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:18:42.713231133 +0000 UTC m=+2278.839166106" watchObservedRunningTime="2026-07-18 01:18:42.713752334 +0000 UTC m=+2278.839687317"
Jul 18 01:18:43.703765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:43.703731    2554 generic.go:356] "Generic (PLEG): container finished" podID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerID="5869fde8581cead83c30c5b191c8474c9c1275ad56e2b029f81d0ec7b07d4191" exitCode=7
Jul 18 01:18:43.704213 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:43.703799    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" event={"ID":"2877c4fc-000f-45fd-b3ff-925e407fcb3f","Type":"ContainerDied","Data":"5869fde8581cead83c30c5b191c8474c9c1275ad56e2b029f81d0ec7b07d4191"}
Jul 18 01:18:43.704213 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:43.704008    2554 scope.go:117] "RemoveContainer" containerID="5869fde8581cead83c30c5b191c8474c9c1275ad56e2b029f81d0ec7b07d4191"
Jul 18 01:18:44.709636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.709579    2554 generic.go:356] "Generic (PLEG): container finished" podID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerID="39c8b91db8f9e7ce311c3167db47ae9445585c5a18e575f39a51c46926bde6b1" exitCode=7
Jul 18 01:18:44.710036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.709653    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" event={"ID":"2877c4fc-000f-45fd-b3ff-925e407fcb3f","Type":"ContainerDied","Data":"39c8b91db8f9e7ce311c3167db47ae9445585c5a18e575f39a51c46926bde6b1"}
Jul 18 01:18:44.710036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.709708    2554 scope.go:117] "RemoveContainer" containerID="5869fde8581cead83c30c5b191c8474c9c1275ad56e2b029f81d0ec7b07d4191"
Jul 18 01:18:44.710036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.709907    2554 scope.go:117] "RemoveContainer" containerID="39c8b91db8f9e7ce311c3167db47ae9445585c5a18e575f39a51c46926bde6b1"
Jul 18 01:18:44.710191 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:44.710125    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth_odh-ai-gateway-infra(2877c4fc-000f-45fd-b3ff-925e407fcb3f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f"
Jul 18 01:18:44.746015 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.745993    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"]
Jul 18 01:18:44.746386 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.746375    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:44.746432 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.746388    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:44.746471 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.746452    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cf8f21-c42d-47f2-af06-31c200a020a5" containerName="revoke-keys"
Jul 18 01:18:44.750601 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.750584    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"
Jul 18 01:18:44.755705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.755683    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"]
Jul 18 01:18:44.939491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:44.939399    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3934f83-3915-450f-b91a-5a92bd6076b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-jbmwx\" (UID: \"a3934f83-3915-450f-b91a-5a92bd6076b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"
Jul 18 01:18:45.040258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:45.040220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3934f83-3915-450f-b91a-5a92bd6076b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-jbmwx\" (UID: \"a3934f83-3915-450f-b91a-5a92bd6076b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"
Jul 18 01:18:45.040881 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:45.040861    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3934f83-3915-450f-b91a-5a92bd6076b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-jbmwx\" (UID: \"a3934f83-3915-450f-b91a-5a92bd6076b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"
Jul 18 01:18:45.062830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:45.062789    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"
Jul 18 01:18:45.397279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:45.397252    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"]
Jul 18 01:18:45.400056 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:18:45.400025    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3934f83_3915_450f_b91a_5a92bd6076b6.slice/crio-5c148d914452c7fc445c3078580cb53a72de8db8772f714e840321103e0ee7be WatchSource:0}: Error finding container 5c148d914452c7fc445c3078580cb53a72de8db8772f714e840321103e0ee7be: Status 404 returned error can't find the container with id 5c148d914452c7fc445c3078580cb53a72de8db8772f714e840321103e0ee7be
Jul 18 01:18:45.715778 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:45.715747    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" event={"ID":"a3934f83-3915-450f-b91a-5a92bd6076b6","Type":"ContainerStarted","Data":"5c148d914452c7fc445c3078580cb53a72de8db8772f714e840321103e0ee7be"}
Jul 18 01:18:45.717694 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:45.717671    2554 scope.go:117] "RemoveContainer" containerID="39c8b91db8f9e7ce311c3167db47ae9445585c5a18e575f39a51c46926bde6b1"
Jul 18 01:18:45.717971 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:45.717948    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth_odh-ai-gateway-infra(2877c4fc-000f-45fd-b3ff-925e407fcb3f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f"
Jul 18 01:18:46.728202 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:46.728096    2554 generic.go:356] "Generic (PLEG): container finished" podID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerID="6d8b78dcc216245bb050523e3059f6bcc9005f855562022ec74b06e52572b500" exitCode=6
Jul 18 01:18:46.728727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:46.728266    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" event={"ID":"a3934f83-3915-450f-b91a-5a92bd6076b6","Type":"ContainerDied","Data":"6d8b78dcc216245bb050523e3059f6bcc9005f855562022ec74b06e52572b500"}
Jul 18 01:18:46.728727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:46.728706    2554 scope.go:117] "RemoveContainer" containerID="6d8b78dcc216245bb050523e3059f6bcc9005f855562022ec74b06e52572b500"
Jul 18 01:18:47.736818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:47.736788    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" event={"ID":"a3934f83-3915-450f-b91a-5a92bd6076b6","Type":"ContainerStarted","Data":"9951315025e05351b0520cf731883e48738646753417bbabda7e9aaf269d94d6"}
Jul 18 01:18:47.753147 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:47.753102    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" podStartSLOduration=3.753087116 podStartE2EDuration="3.753087116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:18:47.750520736 +0000 UTC m=+2283.876455715" watchObservedRunningTime="2026-07-18 01:18:47.753087116 +0000 UTC m=+2283.879022124"
Jul 18 01:18:48.744345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:48.744314    2554 generic.go:356] "Generic (PLEG): container finished" podID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerID="9951315025e05351b0520cf731883e48738646753417bbabda7e9aaf269d94d6" exitCode=6
Jul 18 01:18:48.744812 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:48.744354    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" event={"ID":"a3934f83-3915-450f-b91a-5a92bd6076b6","Type":"ContainerDied","Data":"9951315025e05351b0520cf731883e48738646753417bbabda7e9aaf269d94d6"}
Jul 18 01:18:48.744812 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:48.744399    2554 scope.go:117] "RemoveContainer" containerID="6d8b78dcc216245bb050523e3059f6bcc9005f855562022ec74b06e52572b500"
Jul 18 01:18:48.744812 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:48.744544    2554 scope.go:117] "RemoveContainer" containerID="9951315025e05351b0520cf731883e48738646753417bbabda7e9aaf269d94d6"
Jul 18 01:18:48.744939 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:48.744847    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-jbmwx_odh-ai-gateway-infra(a3934f83-3915-450f-b91a-5a92bd6076b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6"
Jul 18 01:18:49.751535 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:49.751507    2554 scope.go:117] "RemoveContainer" containerID="9951315025e05351b0520cf731883e48738646753417bbabda7e9aaf269d94d6"
Jul 18 01:18:49.751936 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:49.751755    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-jbmwx_odh-ai-gateway-infra(a3934f83-3915-450f-b91a-5a92bd6076b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6"
Jul 18 01:18:53.672805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.672768    2554 patch_prober.go:28] interesting pod/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 18 01:18:53.673222 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.672840    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 18 01:18:53.776145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.776112    2554 generic.go:356] "Generic (PLEG): container finished" podID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerID="93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12" exitCode=0
Jul 18 01:18:53.776301 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.776153    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" event={"ID":"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868","Type":"ContainerDied","Data":"93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12"}
Jul 18 01:18:53.776553 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.776537    2554 scope.go:117] "RemoveContainer" containerID="93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12"
Jul 18 01:18:53.777650 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.777589    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:18:53.777708 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.777692    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:18:53.777747 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:53.777723    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:18:54.783196 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:54.783164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" event={"ID":"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868","Type":"ContainerStarted","Data":"850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be"}
Jul 18 01:18:55.529730 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:55.529689    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:55.529918 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:55.529899    2554 patch_prober.go:28] interesting pod/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 18 01:18:55.529978 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:55.529954    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 18 01:18:56.529979 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.529940    2554 patch_prober.go:28] interesting pod/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 18 01:18:56.530497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.529994    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 18 01:18:56.736148 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.736116    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"]
Jul 18 01:18:56.740123 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.740101    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"
Jul 18 01:18:56.746915 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.746886    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"]
Jul 18 01:18:56.855854 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.855753    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70b21e0-89d8-458b-a630-36a20ba1d34c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq\" (UID: \"e70b21e0-89d8-458b-a630-36a20ba1d34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"
Jul 18 01:18:56.956812 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.956775    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70b21e0-89d8-458b-a630-36a20ba1d34c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq\" (UID: \"e70b21e0-89d8-458b-a630-36a20ba1d34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"
Jul 18 01:18:56.957412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:56.957393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70b21e0-89d8-458b-a630-36a20ba1d34c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq\" (UID: \"e70b21e0-89d8-458b-a630-36a20ba1d34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"
Jul 18 01:18:57.052443 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.052410    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"
Jul 18 01:18:57.230684 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.215410    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"]
Jul 18 01:18:57.233825 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:18:57.233790    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70b21e0_89d8_458b_a630_36a20ba1d34c.slice/crio-aae5c00a98c006a325ee3d82a9cb66ed66a41b250527505bf0d147e053c918e2 WatchSource:0}: Error finding container aae5c00a98c006a325ee3d82a9cb66ed66a41b250527505bf0d147e053c918e2: Status 404 returned error can't find the container with id aae5c00a98c006a325ee3d82a9cb66ed66a41b250527505bf0d147e053c918e2
Jul 18 01:18:57.331397 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.331371    2554 scope.go:117] "RemoveContainer" containerID="39c8b91db8f9e7ce311c3167db47ae9445585c5a18e575f39a51c46926bde6b1"
Jul 18 01:18:57.533705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.530933    2554 patch_prober.go:28] interesting pod/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 18 01:18:57.533705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.530993    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 18 01:18:57.800224 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.800191    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" event={"ID":"2877c4fc-000f-45fd-b3ff-925e407fcb3f","Type":"ContainerStarted","Data":"2be4f3fd52ec7c27ed7366f347fe64512ec9f06466c09e988f3fc7baae3755db"}
Jul 18 01:18:57.802355 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.802259    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" event={"ID":"e70b21e0-89d8-458b-a630-36a20ba1d34c","Type":"ContainerStarted","Data":"e48e745c5bda2f59467d42cba3c3e943b0a66b155dec0d46852e04fa9d268b12"}
Jul 18 01:18:57.802355 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.802302    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" event={"ID":"e70b21e0-89d8-458b-a630-36a20ba1d34c","Type":"ContainerStarted","Data":"aae5c00a98c006a325ee3d82a9cb66ed66a41b250527505bf0d147e053c918e2"}
Jul 18 01:18:57.829320 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:57.829253    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" podStartSLOduration=1.829233471 podStartE2EDuration="1.829233471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:18:57.82914719 +0000 UTC m=+2293.955082171" watchObservedRunningTime="2026-07-18 01:18:57.829233471 +0000 UTC m=+2293.955168454"
Jul 18 01:18:58.529867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.529825    2554 patch_prober.go:28] interesting pod/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 18 01:18:58.530089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.529899    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 18 01:18:58.809357 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.809275    2554 generic.go:356] "Generic (PLEG): container finished" podID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerID="2be4f3fd52ec7c27ed7366f347fe64512ec9f06466c09e988f3fc7baae3755db" exitCode=7
Jul 18 01:18:58.809357 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.809343    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" event={"ID":"2877c4fc-000f-45fd-b3ff-925e407fcb3f","Type":"ContainerDied","Data":"2be4f3fd52ec7c27ed7366f347fe64512ec9f06466c09e988f3fc7baae3755db"}
Jul 18 01:18:58.809889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.809409    2554 scope.go:117] "RemoveContainer" containerID="39c8b91db8f9e7ce311c3167db47ae9445585c5a18e575f39a51c46926bde6b1"
Jul 18 01:18:58.809889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.809710    2554 scope.go:117] "RemoveContainer" containerID="2be4f3fd52ec7c27ed7366f347fe64512ec9f06466c09e988f3fc7baae3755db"
Jul 18 01:18:58.809991 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:58.809950    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth_odh-ai-gateway-infra(2877c4fc-000f-45fd-b3ff-925e407fcb3f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f"
Jul 18 01:18:58.810968 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.810947    2554 generic.go:356] "Generic (PLEG): container finished" podID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerID="e48e745c5bda2f59467d42cba3c3e943b0a66b155dec0d46852e04fa9d268b12" exitCode=6
Jul 18 01:18:58.811066 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.811031    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" event={"ID":"e70b21e0-89d8-458b-a630-36a20ba1d34c","Type":"ContainerDied","Data":"e48e745c5bda2f59467d42cba3c3e943b0a66b155dec0d46852e04fa9d268b12"}
Jul 18 01:18:58.811198 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.811185    2554 scope.go:117] "RemoveContainer" containerID="e48e745c5bda2f59467d42cba3c3e943b0a66b155dec0d46852e04fa9d268b12"
Jul 18 01:18:58.828411 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:58.828390    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"]
Jul 18 01:18:59.533674 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.533651    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:59.534057 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.534037    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:59.534835 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.534817    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:18:59.819508 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.819477    2554 generic.go:356] "Generic (PLEG): container finished" podID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerID="b0af49dcf6627553bd4fe9448f8db24a6e6feda6aca278c23a248b32b499c3ce" exitCode=6
Jul 18 01:18:59.819893 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.819551    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" event={"ID":"e70b21e0-89d8-458b-a630-36a20ba1d34c","Type":"ContainerDied","Data":"b0af49dcf6627553bd4fe9448f8db24a6e6feda6aca278c23a248b32b499c3ce"}
Jul 18 01:18:59.819893 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.819631    2554 scope.go:117] "RemoveContainer" containerID="e48e745c5bda2f59467d42cba3c3e943b0a66b155dec0d46852e04fa9d268b12"
Jul 18 01:18:59.819893 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.819858    2554 scope.go:117] "RemoveContainer" containerID="b0af49dcf6627553bd4fe9448f8db24a6e6feda6aca278c23a248b32b499c3ce"
Jul 18 01:18:59.820100 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:18:59.820077    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq_odh-ai-gateway-infra(e70b21e0-89d8-458b-a630-36a20ba1d34c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c"
Jul 18 01:18:59.958532 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:18:59.958509    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"
Jul 18 01:19:00.085358 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.085255    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2877c4fc-000f-45fd-b3ff-925e407fcb3f-maas-api-service-ca\") pod \"2877c4fc-000f-45fd-b3ff-925e407fcb3f\" (UID: \"2877c4fc-000f-45fd-b3ff-925e407fcb3f\") "
Jul 18 01:19:00.085600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.085577    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2877c4fc-000f-45fd-b3ff-925e407fcb3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2877c4fc-000f-45fd-b3ff-925e407fcb3f" (UID: "2877c4fc-000f-45fd-b3ff-925e407fcb3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:19:00.186377 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.186339    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2877c4fc-000f-45fd-b3ff-925e407fcb3f-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:19:00.826272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.826243    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"
Jul 18 01:19:00.826793 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.826270    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth" event={"ID":"2877c4fc-000f-45fd-b3ff-925e407fcb3f","Type":"ContainerDied","Data":"4b912aa99e4e933f05aaa7daab56dab88b472f3f734cc49808d28706b0277cc6"}
Jul 18 01:19:00.826793 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.826338    2554 scope.go:117] "RemoveContainer" containerID="2be4f3fd52ec7c27ed7366f347fe64512ec9f06466c09e988f3fc7baae3755db"
Jul 18 01:19:00.828307 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.828141    2554 scope.go:117] "RemoveContainer" containerID="b0af49dcf6627553bd4fe9448f8db24a6e6feda6aca278c23a248b32b499c3ce"
Jul 18 01:19:00.828423 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:00.828376    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq_odh-ai-gateway-infra(e70b21e0-89d8-458b-a630-36a20ba1d34c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c"
Jul 18 01:19:00.842066 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.842042    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"]
Jul 18 01:19:00.848764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:00.848745    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-stcth"]
Jul 18 01:19:02.336195 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:02.336163    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" path="/var/lib/kubelet/pods/2877c4fc-000f-45fd-b3ff-925e407fcb3f/volumes"
Jul 18 01:19:03.331035 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:03.331004    2554 scope.go:117] "RemoveContainer" containerID="9951315025e05351b0520cf731883e48738646753417bbabda7e9aaf269d94d6"
Jul 18 01:19:03.846287 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:03.846246    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" event={"ID":"a3934f83-3915-450f-b91a-5a92bd6076b6","Type":"ContainerStarted","Data":"76f88e400543061c21e89bd871eaf963d1e35414205fe576d54e486fcac52946"}
Jul 18 01:19:04.852410 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:04.852373    2554 generic.go:356] "Generic (PLEG): container finished" podID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerID="76f88e400543061c21e89bd871eaf963d1e35414205fe576d54e486fcac52946" exitCode=6
Jul 18 01:19:04.852826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:04.852447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" event={"ID":"a3934f83-3915-450f-b91a-5a92bd6076b6","Type":"ContainerDied","Data":"76f88e400543061c21e89bd871eaf963d1e35414205fe576d54e486fcac52946"}
Jul 18 01:19:04.852826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:04.852507    2554 scope.go:117] "RemoveContainer" containerID="9951315025e05351b0520cf731883e48738646753417bbabda7e9aaf269d94d6"
Jul 18 01:19:04.852826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:04.852806    2554 scope.go:117] "RemoveContainer" containerID="76f88e400543061c21e89bd871eaf963d1e35414205fe576d54e486fcac52946"
Jul 18 01:19:04.853058 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:04.853036    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-jbmwx_odh-ai-gateway-infra(a3934f83-3915-450f-b91a-5a92bd6076b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6"
Jul 18 01:19:04.876740 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:04.876715    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"]
Jul 18 01:19:05.999251 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:05.999225    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"
Jul 18 01:19:06.141694 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.141581    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3934f83-3915-450f-b91a-5a92bd6076b6-maas-api-service-ca\") pod \"a3934f83-3915-450f-b91a-5a92bd6076b6\" (UID: \"a3934f83-3915-450f-b91a-5a92bd6076b6\") "
Jul 18 01:19:06.142010 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.141989    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3934f83-3915-450f-b91a-5a92bd6076b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3934f83-3915-450f-b91a-5a92bd6076b6" (UID: "a3934f83-3915-450f-b91a-5a92bd6076b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:19:06.243044 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.242995    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3934f83-3915-450f-b91a-5a92bd6076b6-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:19:06.865089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.865061    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"
Jul 18 01:19:06.865272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.865057    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx" event={"ID":"a3934f83-3915-450f-b91a-5a92bd6076b6","Type":"ContainerDied","Data":"5c148d914452c7fc445c3078580cb53a72de8db8772f714e840321103e0ee7be"}
Jul 18 01:19:06.865411 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.865307    2554 scope.go:117] "RemoveContainer" containerID="76f88e400543061c21e89bd871eaf963d1e35414205fe576d54e486fcac52946"
Jul 18 01:19:06.880791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.880770    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"]
Jul 18 01:19:06.888965 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:06.888945    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-jbmwx"]
Jul 18 01:19:08.336309 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:08.336274    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" path="/var/lib/kubelet/pods/a3934f83-3915-450f-b91a-5a92bd6076b6/volumes"
Jul 18 01:19:12.330343 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:12.330308    2554 scope.go:117] "RemoveContainer" containerID="b0af49dcf6627553bd4fe9448f8db24a6e6feda6aca278c23a248b32b499c3ce"
Jul 18 01:19:12.902847 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:12.902815    2554 generic.go:356] "Generic (PLEG): container finished" podID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerID="e73eb1d0bbf0ecbadc275561949f6495138346f17a44621afb89d042a93b61e1" exitCode=6
Jul 18 01:19:12.902984 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:12.902888    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" event={"ID":"e70b21e0-89d8-458b-a630-36a20ba1d34c","Type":"ContainerDied","Data":"e73eb1d0bbf0ecbadc275561949f6495138346f17a44621afb89d042a93b61e1"}
Jul 18 01:19:12.902984 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:12.902946    2554 scope.go:117] "RemoveContainer" containerID="b0af49dcf6627553bd4fe9448f8db24a6e6feda6aca278c23a248b32b499c3ce"
Jul 18 01:19:12.903239 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:12.903220    2554 scope.go:117] "RemoveContainer" containerID="e73eb1d0bbf0ecbadc275561949f6495138346f17a44621afb89d042a93b61e1"
Jul 18 01:19:12.903513 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:12.903491    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq_odh-ai-gateway-infra(e70b21e0-89d8-458b-a630-36a20ba1d34c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c"
Jul 18 01:19:13.928103 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:13.928063    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"]
Jul 18 01:19:14.064344 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.064320    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"
Jul 18 01:19:14.219587 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.219499    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70b21e0-89d8-458b-a630-36a20ba1d34c-maas-api-service-ca\") pod \"e70b21e0-89d8-458b-a630-36a20ba1d34c\" (UID: \"e70b21e0-89d8-458b-a630-36a20ba1d34c\") "
Jul 18 01:19:14.219872 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.219849    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70b21e0-89d8-458b-a630-36a20ba1d34c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e70b21e0-89d8-458b-a630-36a20ba1d34c" (UID: "e70b21e0-89d8-458b-a630-36a20ba1d34c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:19:14.320638 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.320583    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70b21e0-89d8-458b-a630-36a20ba1d34c-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:19:14.916131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.916101    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"
Jul 18 01:19:14.916325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.916100    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq" event={"ID":"e70b21e0-89d8-458b-a630-36a20ba1d34c","Type":"ContainerDied","Data":"aae5c00a98c006a325ee3d82a9cb66ed66a41b250527505bf0d147e053c918e2"}
Jul 18 01:19:14.916325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.916229    2554 scope.go:117] "RemoveContainer" containerID="e73eb1d0bbf0ecbadc275561949f6495138346f17a44621afb89d042a93b61e1"
Jul 18 01:19:14.931144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.931120    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"]
Jul 18 01:19:14.936716 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:14.936694    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-n8pvq"]
Jul 18 01:19:16.336001 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:16.335967    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" path="/var/lib/kubelet/pods/e70b21e0-89d8-458b-a630-36a20ba1d34c/volumes"
Jul 18 01:19:18.718150 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718113    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"]
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718497    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718508    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718529    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718534    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718540    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718546    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718555    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718560    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718567    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718573    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718581    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718586    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718594    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.718611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718598    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718674    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718683    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718688    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718693    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718699    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718705    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718711    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718716    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70b21e0-89d8-458b-a630-36a20ba1d34c" containerName="revoke-keys"
Jul 18 01:19:18.719217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.718721    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:18.722773 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.722756    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"
Jul 18 01:19:18.725665 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.725641    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:19:18.728103 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.728079    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"]
Jul 18 01:19:18.869165 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.869129    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44907bd-f707-4109-a375-d9b1435bf2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn\" (UID: \"f44907bd-f707-4109-a375-d9b1435bf2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"
Jul 18 01:19:18.969824 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.969730    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44907bd-f707-4109-a375-d9b1435bf2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn\" (UID: \"f44907bd-f707-4109-a375-d9b1435bf2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"
Jul 18 01:19:18.970330 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:18.970310    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44907bd-f707-4109-a375-d9b1435bf2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn\" (UID: \"f44907bd-f707-4109-a375-d9b1435bf2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"
Jul 18 01:19:19.035301 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:19.035268    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"
Jul 18 01:19:19.170053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:19.170021    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"]
Jul 18 01:19:19.171999 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:19:19.171970    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44907bd_f707_4109_a375_d9b1435bf2c4.slice/crio-eac1c1bcfa34b1958776007a1cb3c3bf65159a988c50d0f7fafb743708972af0 WatchSource:0}: Error finding container eac1c1bcfa34b1958776007a1cb3c3bf65159a988c50d0f7fafb743708972af0: Status 404 returned error can't find the container with id eac1c1bcfa34b1958776007a1cb3c3bf65159a988c50d0f7fafb743708972af0
Jul 18 01:19:19.942936 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:19.942903    2554 generic.go:356] "Generic (PLEG): container finished" podID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerID="4970890c704b9284c4f712803b791a55f80a792d47dd4a1fdf3a851cbe7ae2e5" exitCode=6
Jul 18 01:19:19.943362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:19.942976    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" event={"ID":"f44907bd-f707-4109-a375-d9b1435bf2c4","Type":"ContainerDied","Data":"4970890c704b9284c4f712803b791a55f80a792d47dd4a1fdf3a851cbe7ae2e5"}
Jul 18 01:19:19.943362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:19.943029    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" event={"ID":"f44907bd-f707-4109-a375-d9b1435bf2c4","Type":"ContainerStarted","Data":"eac1c1bcfa34b1958776007a1cb3c3bf65159a988c50d0f7fafb743708972af0"}
Jul 18 01:19:19.943362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:19.943258    2554 scope.go:117] "RemoveContainer" containerID="4970890c704b9284c4f712803b791a55f80a792d47dd4a1fdf3a851cbe7ae2e5"
Jul 18 01:19:20.950121 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:20.950037    2554 generic.go:356] "Generic (PLEG): container finished" podID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerID="6be6bef4c7cea8c644b2bb533fa0942fbce04195d17f7f47677cf946a5047386" exitCode=6
Jul 18 01:19:20.950578 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:20.950115    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" event={"ID":"f44907bd-f707-4109-a375-d9b1435bf2c4","Type":"ContainerDied","Data":"6be6bef4c7cea8c644b2bb533fa0942fbce04195d17f7f47677cf946a5047386"}
Jul 18 01:19:20.950578 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:20.950174    2554 scope.go:117] "RemoveContainer" containerID="4970890c704b9284c4f712803b791a55f80a792d47dd4a1fdf3a851cbe7ae2e5"
Jul 18 01:19:20.950578 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:20.950394    2554 scope.go:117] "RemoveContainer" containerID="6be6bef4c7cea8c644b2bb533fa0942fbce04195d17f7f47677cf946a5047386"
Jul 18 01:19:20.950778 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:20.950721    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn_odh-ai-gateway-infra(f44907bd-f707-4109-a375-d9b1435bf2c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4"
Jul 18 01:19:21.956587 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:21.956557    2554 scope.go:117] "RemoveContainer" containerID="6be6bef4c7cea8c644b2bb533fa0942fbce04195d17f7f47677cf946a5047386"
Jul 18 01:19:21.957029 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:21.956812    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn_odh-ai-gateway-infra(f44907bd-f707-4109-a375-d9b1435bf2c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4"
Jul 18 01:19:31.867895 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.867861    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"]
Jul 18 01:19:31.868322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.868282    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:31.868322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.868294    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2877c4fc-000f-45fd-b3ff-925e407fcb3f" containerName="revoke-keys"
Jul 18 01:19:31.868322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.868307    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:31.868322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.868312    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3934f83-3915-450f-b91a-5a92bd6076b6" containerName="revoke-keys"
Jul 18 01:19:31.872699 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.872680    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"
Jul 18 01:19:31.879264 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.879234    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"]
Jul 18 01:19:31.910173 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:31.900803    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9d96-c21e-47aa-81b8-d5961b2009cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx\" (UID: \"a5ed9d96-c21e-47aa-81b8-d5961b2009cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"
Jul 18 01:19:32.001502 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:32.001462    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9d96-c21e-47aa-81b8-d5961b2009cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx\" (UID: \"a5ed9d96-c21e-47aa-81b8-d5961b2009cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"
Jul 18 01:19:32.002073 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:32.002054    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9d96-c21e-47aa-81b8-d5961b2009cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx\" (UID: \"a5ed9d96-c21e-47aa-81b8-d5961b2009cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"
Jul 18 01:19:32.185642 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:32.185524    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"
Jul 18 01:19:32.522591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:32.522518    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"]
Jul 18 01:19:32.525188 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:19:32.525154    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ed9d96_c21e_47aa_81b8_d5961b2009cb.slice/crio-7945201693e7d4aa9529776711d0680f2a0c98e9713a7d154dbdbeb9556e9ee3 WatchSource:0}: Error finding container 7945201693e7d4aa9529776711d0680f2a0c98e9713a7d154dbdbeb9556e9ee3: Status 404 returned error can't find the container with id 7945201693e7d4aa9529776711d0680f2a0c98e9713a7d154dbdbeb9556e9ee3
Jul 18 01:19:33.012325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:33.012285    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" event={"ID":"a5ed9d96-c21e-47aa-81b8-d5961b2009cb","Type":"ContainerStarted","Data":"23ead7058866372498f4e246d5e3248622eb88fa1caaf02d27cb0923d6d37f4e"}
Jul 18 01:19:33.012714 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:33.012354    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" event={"ID":"a5ed9d96-c21e-47aa-81b8-d5961b2009cb","Type":"ContainerStarted","Data":"7945201693e7d4aa9529776711d0680f2a0c98e9713a7d154dbdbeb9556e9ee3"}
Jul 18 01:19:33.028272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:33.028223    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" podStartSLOduration=2.028210242 podStartE2EDuration="2.028210242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:19:33.02665271 +0000 UTC m=+2329.152587694" watchObservedRunningTime="2026-07-18 01:19:33.028210242 +0000 UTC m=+2329.154145279"
Jul 18 01:19:34.018940 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:34.018901    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerID="23ead7058866372498f4e246d5e3248622eb88fa1caaf02d27cb0923d6d37f4e" exitCode=7
Jul 18 01:19:34.019393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:34.018981    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" event={"ID":"a5ed9d96-c21e-47aa-81b8-d5961b2009cb","Type":"ContainerDied","Data":"23ead7058866372498f4e246d5e3248622eb88fa1caaf02d27cb0923d6d37f4e"}
Jul 18 01:19:34.019393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:34.019143    2554 scope.go:117] "RemoveContainer" containerID="23ead7058866372498f4e246d5e3248622eb88fa1caaf02d27cb0923d6d37f4e"
Jul 18 01:19:35.026349 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:35.026313    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerID="7dcb45e28ed2615903017a4d588f936320205d0d1b10765b2eb513038c85d14a" exitCode=7
Jul 18 01:19:35.026780 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:35.026369    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" event={"ID":"a5ed9d96-c21e-47aa-81b8-d5961b2009cb","Type":"ContainerDied","Data":"7dcb45e28ed2615903017a4d588f936320205d0d1b10765b2eb513038c85d14a"}
Jul 18 01:19:35.026780 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:35.026421    2554 scope.go:117] "RemoveContainer" containerID="23ead7058866372498f4e246d5e3248622eb88fa1caaf02d27cb0923d6d37f4e"
Jul 18 01:19:35.026780 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:35.026598    2554 scope.go:117] "RemoveContainer" containerID="7dcb45e28ed2615903017a4d588f936320205d0d1b10765b2eb513038c85d14a"
Jul 18 01:19:35.026906 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:35.026877    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx_odh-ai-gateway-infra(a5ed9d96-c21e-47aa-81b8-d5961b2009cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb"
Jul 18 01:19:36.032870 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:36.032839    2554 scope.go:117] "RemoveContainer" containerID="7dcb45e28ed2615903017a4d588f936320205d0d1b10765b2eb513038c85d14a"
Jul 18 01:19:36.033259 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:36.033053    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx_odh-ai-gateway-infra(a5ed9d96-c21e-47aa-81b8-d5961b2009cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb"
Jul 18 01:19:37.331260 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:37.331233    2554 scope.go:117] "RemoveContainer" containerID="6be6bef4c7cea8c644b2bb533fa0942fbce04195d17f7f47677cf946a5047386"
Jul 18 01:19:38.045131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:38.045097    2554 generic.go:356] "Generic (PLEG): container finished" podID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerID="77e807d4c601636ef0edc6cf1a6a311bc803c9dbb334052feebc11d1a0acc2ed" exitCode=6
Jul 18 01:19:38.045305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:38.045178    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" event={"ID":"f44907bd-f707-4109-a375-d9b1435bf2c4","Type":"ContainerDied","Data":"77e807d4c601636ef0edc6cf1a6a311bc803c9dbb334052feebc11d1a0acc2ed"}
Jul 18 01:19:38.045305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:38.045275    2554 scope.go:117] "RemoveContainer" containerID="6be6bef4c7cea8c644b2bb533fa0942fbce04195d17f7f47677cf946a5047386"
Jul 18 01:19:38.045573 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:38.045550    2554 scope.go:117] "RemoveContainer" containerID="77e807d4c601636ef0edc6cf1a6a311bc803c9dbb334052feebc11d1a0acc2ed"
Jul 18 01:19:38.045843 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:38.045820    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn_odh-ai-gateway-infra(f44907bd-f707-4109-a375-d9b1435bf2c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4"
Jul 18 01:19:39.068408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:39.068373    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"]
Jul 18 01:19:39.209939 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:39.209911    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"
Jul 18 01:19:39.275408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:39.275381    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44907bd-f707-4109-a375-d9b1435bf2c4-maas-api-service-ca\") pod \"f44907bd-f707-4109-a375-d9b1435bf2c4\" (UID: \"f44907bd-f707-4109-a375-d9b1435bf2c4\") "
Jul 18 01:19:39.275763 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:39.275736    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44907bd-f707-4109-a375-d9b1435bf2c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f44907bd-f707-4109-a375-d9b1435bf2c4" (UID: "f44907bd-f707-4109-a375-d9b1435bf2c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:19:39.376522 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:39.376438    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f44907bd-f707-4109-a375-d9b1435bf2c4-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:19:40.058265 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:40.058235    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn" event={"ID":"f44907bd-f707-4109-a375-d9b1435bf2c4","Type":"ContainerDied","Data":"eac1c1bcfa34b1958776007a1cb3c3bf65159a988c50d0f7fafb743708972af0"}
Jul 18 01:19:40.058472 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:40.058297    2554 scope.go:117] "RemoveContainer" containerID="77e807d4c601636ef0edc6cf1a6a311bc803c9dbb334052feebc11d1a0acc2ed"
Jul 18 01:19:40.058472 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:40.058243    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"
Jul 18 01:19:40.080145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:40.080122    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"]
Jul 18 01:19:40.084216 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:40.084195    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-s6cjn"]
Jul 18 01:19:40.336569 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:40.336491    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" path="/var/lib/kubelet/pods/f44907bd-f707-4109-a375-d9b1435bf2c4/volumes"
Jul 18 01:19:44.790702 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.790665    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"]
Jul 18 01:19:44.791107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.791059    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:44.791107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.791070    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:44.791107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.791080    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:44.791107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.791086    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:44.791236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.791156    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:44.791236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.791165    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:44.795779 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.795760    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"
Jul 18 01:19:44.800384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.800354    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"]
Jul 18 01:19:44.938499 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:44.938408    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4214e923-0269-4301-adff-3c8926d2ff45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gwxn4\" (UID: \"4214e923-0269-4301-adff-3c8926d2ff45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"
Jul 18 01:19:45.039562 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:45.039525    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4214e923-0269-4301-adff-3c8926d2ff45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gwxn4\" (UID: \"4214e923-0269-4301-adff-3c8926d2ff45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"
Jul 18 01:19:45.040180 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:45.040159    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4214e923-0269-4301-adff-3c8926d2ff45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gwxn4\" (UID: \"4214e923-0269-4301-adff-3c8926d2ff45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"
Jul 18 01:19:45.108302 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:45.108271    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"
Jul 18 01:19:45.240326 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:45.240300    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"]
Jul 18 01:19:45.242680 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:19:45.242652    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4214e923_0269_4301_adff_3c8926d2ff45.slice/crio-8c455674fceb50a4fb96dbe4ab614d8dc34754b91d65a3d551ec3d8fde537e86 WatchSource:0}: Error finding container 8c455674fceb50a4fb96dbe4ab614d8dc34754b91d65a3d551ec3d8fde537e86: Status 404 returned error can't find the container with id 8c455674fceb50a4fb96dbe4ab614d8dc34754b91d65a3d551ec3d8fde537e86
Jul 18 01:19:46.091258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.091174    2554 generic.go:356] "Generic (PLEG): container finished" podID="4214e923-0269-4301-adff-3c8926d2ff45" containerID="d461a50aa5b52e8e40f740fd1289664217153af564baff46b49caa00ab25a5ab" exitCode=6
Jul 18 01:19:46.091258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.091226    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" event={"ID":"4214e923-0269-4301-adff-3c8926d2ff45","Type":"ContainerDied","Data":"d461a50aa5b52e8e40f740fd1289664217153af564baff46b49caa00ab25a5ab"}
Jul 18 01:19:46.091708 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.091267    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" event={"ID":"4214e923-0269-4301-adff-3c8926d2ff45","Type":"ContainerStarted","Data":"8c455674fceb50a4fb96dbe4ab614d8dc34754b91d65a3d551ec3d8fde537e86"}
Jul 18 01:19:46.091708 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.091485    2554 scope.go:117] "RemoveContainer" containerID="d461a50aa5b52e8e40f740fd1289664217153af564baff46b49caa00ab25a5ab"
Jul 18 01:19:46.771195 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.771161    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"]
Jul 18 01:19:46.771578 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.771564    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:46.771645 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.771580    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:46.771682 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.771670    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f44907bd-f707-4109-a375-d9b1435bf2c4" containerName="revoke-keys"
Jul 18 01:19:46.774713 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.774697    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"
Jul 18 01:19:46.781239 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.781215    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"]
Jul 18 01:19:46.855147 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.855114    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b869c385-b17d-4be3-b3ad-f5ba7e5eac48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk\" (UID: \"b869c385-b17d-4be3-b3ad-f5ba7e5eac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"
Jul 18 01:19:46.956323 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.956283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b869c385-b17d-4be3-b3ad-f5ba7e5eac48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk\" (UID: \"b869c385-b17d-4be3-b3ad-f5ba7e5eac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"
Jul 18 01:19:46.956946 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:46.956925    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b869c385-b17d-4be3-b3ad-f5ba7e5eac48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk\" (UID: \"b869c385-b17d-4be3-b3ad-f5ba7e5eac48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"
Jul 18 01:19:47.086234 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:47.086150    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"
Jul 18 01:19:47.098866 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:47.098839    2554 generic.go:356] "Generic (PLEG): container finished" podID="4214e923-0269-4301-adff-3c8926d2ff45" containerID="5354164f7591daa44eddc37b06c8bc1cfc4f36e2ebebc5bd24172233dc09727e" exitCode=6
Jul 18 01:19:47.099261 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:47.098929    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" event={"ID":"4214e923-0269-4301-adff-3c8926d2ff45","Type":"ContainerDied","Data":"5354164f7591daa44eddc37b06c8bc1cfc4f36e2ebebc5bd24172233dc09727e"}
Jul 18 01:19:47.099261 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:47.099009    2554 scope.go:117] "RemoveContainer" containerID="d461a50aa5b52e8e40f740fd1289664217153af564baff46b49caa00ab25a5ab"
Jul 18 01:19:47.099261 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:47.099230    2554 scope.go:117] "RemoveContainer" containerID="5354164f7591daa44eddc37b06c8bc1cfc4f36e2ebebc5bd24172233dc09727e"
Jul 18 01:19:47.099521 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:47.099498    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gwxn4_odh-ai-gateway-infra(4214e923-0269-4301-adff-3c8926d2ff45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" podUID="4214e923-0269-4301-adff-3c8926d2ff45"
Jul 18 01:19:47.219127 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:47.219099    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"]
Jul 18 01:19:47.221008 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:19:47.220976    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb869c385_b17d_4be3_b3ad_f5ba7e5eac48.slice/crio-508756749a9887021e74e88687cffec7c471c9863b8cf9a7abc7cdd3de9e9d97 WatchSource:0}: Error finding container 508756749a9887021e74e88687cffec7c471c9863b8cf9a7abc7cdd3de9e9d97: Status 404 returned error can't find the container with id 508756749a9887021e74e88687cffec7c471c9863b8cf9a7abc7cdd3de9e9d97
Jul 18 01:19:48.112816 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:48.112782    2554 generic.go:356] "Generic (PLEG): container finished" podID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerID="711b864572c74e03919ef11f56e07d8ce5b027ab943f7b8241c30918185bf43f" exitCode=6
Jul 18 01:19:48.113311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:48.112865    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" event={"ID":"b869c385-b17d-4be3-b3ad-f5ba7e5eac48","Type":"ContainerDied","Data":"711b864572c74e03919ef11f56e07d8ce5b027ab943f7b8241c30918185bf43f"}
Jul 18 01:19:48.113311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:48.112951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" event={"ID":"b869c385-b17d-4be3-b3ad-f5ba7e5eac48","Type":"ContainerStarted","Data":"508756749a9887021e74e88687cffec7c471c9863b8cf9a7abc7cdd3de9e9d97"}
Jul 18 01:19:48.113311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:48.113070    2554 scope.go:117] "RemoveContainer" containerID="711b864572c74e03919ef11f56e07d8ce5b027ab943f7b8241c30918185bf43f"
Jul 18 01:19:48.114687 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:48.114669    2554 scope.go:117] "RemoveContainer" containerID="5354164f7591daa44eddc37b06c8bc1cfc4f36e2ebebc5bd24172233dc09727e"
Jul 18 01:19:48.114878 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:48.114859    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gwxn4_odh-ai-gateway-infra(4214e923-0269-4301-adff-3c8926d2ff45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" podUID="4214e923-0269-4301-adff-3c8926d2ff45"
Jul 18 01:19:49.121322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:49.121290    2554 generic.go:356] "Generic (PLEG): container finished" podID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerID="8dafe52ca0227e58a5164245f54510964bf62851ca549fcaaacfea0a045935ff" exitCode=6
Jul 18 01:19:49.121889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:49.121374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" event={"ID":"b869c385-b17d-4be3-b3ad-f5ba7e5eac48","Type":"ContainerDied","Data":"8dafe52ca0227e58a5164245f54510964bf62851ca549fcaaacfea0a045935ff"}
Jul 18 01:19:49.121889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:49.121444    2554 scope.go:117] "RemoveContainer" containerID="711b864572c74e03919ef11f56e07d8ce5b027ab943f7b8241c30918185bf43f"
Jul 18 01:19:49.121889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:49.121565    2554 scope.go:117] "RemoveContainer" containerID="8dafe52ca0227e58a5164245f54510964bf62851ca549fcaaacfea0a045935ff"
Jul 18 01:19:49.121889 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:49.121815    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk_odh-ai-gateway-infra(b869c385-b17d-4be3-b3ad-f5ba7e5eac48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48"
Jul 18 01:19:49.330336 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:49.330310    2554 scope.go:117] "RemoveContainer" containerID="7dcb45e28ed2615903017a4d588f936320205d0d1b10765b2eb513038c85d14a"
Jul 18 01:19:50.128907 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:50.128881    2554 scope.go:117] "RemoveContainer" containerID="8dafe52ca0227e58a5164245f54510964bf62851ca549fcaaacfea0a045935ff"
Jul 18 01:19:50.129374 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:50.129138    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk_odh-ai-gateway-infra(b869c385-b17d-4be3-b3ad-f5ba7e5eac48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48"
Jul 18 01:19:50.130285 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:50.130264    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerID="c53761d3ac08c62695dbc4e133760f906d83d6ef5e43d5336385e309bc6ca62e" exitCode=7
Jul 18 01:19:50.130404 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:50.130336    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" event={"ID":"a5ed9d96-c21e-47aa-81b8-d5961b2009cb","Type":"ContainerDied","Data":"c53761d3ac08c62695dbc4e133760f906d83d6ef5e43d5336385e309bc6ca62e"}
Jul 18 01:19:50.130404 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:50.130390    2554 scope.go:117] "RemoveContainer" containerID="7dcb45e28ed2615903017a4d588f936320205d0d1b10765b2eb513038c85d14a"
Jul 18 01:19:50.130653 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:50.130609    2554 scope.go:117] "RemoveContainer" containerID="c53761d3ac08c62695dbc4e133760f906d83d6ef5e43d5336385e309bc6ca62e"
Jul 18 01:19:50.130837 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:19:50.130820    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx_odh-ai-gateway-infra(a5ed9d96-c21e-47aa-81b8-d5961b2009cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb"
Jul 18 01:19:51.167247 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:51.167214    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"]
Jul 18 01:19:51.302060 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:51.302034    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"
Jul 18 01:19:51.403022 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:51.402994    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9d96-c21e-47aa-81b8-d5961b2009cb-maas-api-service-ca\") pod \"a5ed9d96-c21e-47aa-81b8-d5961b2009cb\" (UID: \"a5ed9d96-c21e-47aa-81b8-d5961b2009cb\") "
Jul 18 01:19:51.403349 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:51.403322    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ed9d96-c21e-47aa-81b8-d5961b2009cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5ed9d96-c21e-47aa-81b8-d5961b2009cb" (UID: "a5ed9d96-c21e-47aa-81b8-d5961b2009cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:19:51.504051 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:51.504018    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed9d96-c21e-47aa-81b8-d5961b2009cb-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:19:52.142957 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:52.142915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx" event={"ID":"a5ed9d96-c21e-47aa-81b8-d5961b2009cb","Type":"ContainerDied","Data":"7945201693e7d4aa9529776711d0680f2a0c98e9713a7d154dbdbeb9556e9ee3"}
Jul 18 01:19:52.143144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:52.142979    2554 scope.go:117] "RemoveContainer" containerID="c53761d3ac08c62695dbc4e133760f906d83d6ef5e43d5336385e309bc6ca62e"
Jul 18 01:19:52.143144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:52.142933    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"
Jul 18 01:19:52.163655 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:52.163626    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"]
Jul 18 01:19:52.166950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:52.166927    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-67kjx"]
Jul 18 01:19:52.341118 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:52.341084    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" path="/var/lib/kubelet/pods/a5ed9d96-c21e-47aa-81b8-d5961b2009cb/volumes"
Jul 18 01:19:59.331135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:19:59.331107    2554 scope.go:117] "RemoveContainer" containerID="5354164f7591daa44eddc37b06c8bc1cfc4f36e2ebebc5bd24172233dc09727e"
Jul 18 01:20:00.192135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:00.192051    2554 generic.go:356] "Generic (PLEG): container finished" podID="4214e923-0269-4301-adff-3c8926d2ff45" containerID="2137ac47712b64f3e0f0113e3f59e75061540c48f19662ea29edaa9eaad2f6b9" exitCode=6
Jul 18 01:20:00.192135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:00.192098    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" event={"ID":"4214e923-0269-4301-adff-3c8926d2ff45","Type":"ContainerDied","Data":"2137ac47712b64f3e0f0113e3f59e75061540c48f19662ea29edaa9eaad2f6b9"}
Jul 18 01:20:00.192326 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:00.192167    2554 scope.go:117] "RemoveContainer" containerID="5354164f7591daa44eddc37b06c8bc1cfc4f36e2ebebc5bd24172233dc09727e"
Jul 18 01:20:00.192456 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:00.192435    2554 scope.go:117] "RemoveContainer" containerID="2137ac47712b64f3e0f0113e3f59e75061540c48f19662ea29edaa9eaad2f6b9"
Jul 18 01:20:00.192680 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:00.192657    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gwxn4_odh-ai-gateway-infra(4214e923-0269-4301-adff-3c8926d2ff45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" podUID="4214e923-0269-4301-adff-3c8926d2ff45"
Jul 18 01:20:01.218546 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:01.218513    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"]
Jul 18 01:20:01.330572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:01.330543    2554 scope.go:117] "RemoveContainer" containerID="8dafe52ca0227e58a5164245f54510964bf62851ca549fcaaacfea0a045935ff"
Jul 18 01:20:01.378186 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:01.378163    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"
Jul 18 01:20:01.394919 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:01.394893    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4214e923-0269-4301-adff-3c8926d2ff45-maas-api-service-ca\") pod \"4214e923-0269-4301-adff-3c8926d2ff45\" (UID: \"4214e923-0269-4301-adff-3c8926d2ff45\") "
Jul 18 01:20:01.395323 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:01.395289    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4214e923-0269-4301-adff-3c8926d2ff45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4214e923-0269-4301-adff-3c8926d2ff45" (UID: "4214e923-0269-4301-adff-3c8926d2ff45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:20:01.396352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:01.396326    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4214e923-0269-4301-adff-3c8926d2ff45-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:20:02.205126 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.205091    2554 generic.go:356] "Generic (PLEG): container finished" podID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerID="e8566454c72c8e8818a085a9cd9ceaef9c577b0204496d54d63ea0d6ca22fe59" exitCode=6
Jul 18 01:20:02.205282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.205159    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" event={"ID":"b869c385-b17d-4be3-b3ad-f5ba7e5eac48","Type":"ContainerDied","Data":"e8566454c72c8e8818a085a9cd9ceaef9c577b0204496d54d63ea0d6ca22fe59"}
Jul 18 01:20:02.205282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.205213    2554 scope.go:117] "RemoveContainer" containerID="8dafe52ca0227e58a5164245f54510964bf62851ca549fcaaacfea0a045935ff"
Jul 18 01:20:02.205491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.205470    2554 scope.go:117] "RemoveContainer" containerID="e8566454c72c8e8818a085a9cd9ceaef9c577b0204496d54d63ea0d6ca22fe59"
Jul 18 01:20:02.205768 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:02.205743    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk_odh-ai-gateway-infra(b869c385-b17d-4be3-b3ad-f5ba7e5eac48)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48"
Jul 18 01:20:02.206582 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.206563    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"
Jul 18 01:20:02.206709 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.206589    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4" event={"ID":"4214e923-0269-4301-adff-3c8926d2ff45","Type":"ContainerDied","Data":"8c455674fceb50a4fb96dbe4ab614d8dc34754b91d65a3d551ec3d8fde537e86"}
Jul 18 01:20:02.219906 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.219649    2554 scope.go:117] "RemoveContainer" containerID="2137ac47712b64f3e0f0113e3f59e75061540c48f19662ea29edaa9eaad2f6b9"
Jul 18 01:20:02.236574 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.236552    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"]
Jul 18 01:20:02.240222 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.240201    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gwxn4"]
Jul 18 01:20:02.336071 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:02.336004    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4214e923-0269-4301-adff-3c8926d2ff45" path="/var/lib/kubelet/pods/4214e923-0269-4301-adff-3c8926d2ff45/volumes"
Jul 18 01:20:03.228598 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:03.228561    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"]
Jul 18 01:20:03.366225 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:03.366201    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"
Jul 18 01:20:03.414947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:03.414913    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b869c385-b17d-4be3-b3ad-f5ba7e5eac48-maas-api-service-ca\") pod \"b869c385-b17d-4be3-b3ad-f5ba7e5eac48\" (UID: \"b869c385-b17d-4be3-b3ad-f5ba7e5eac48\") "
Jul 18 01:20:03.415338 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:03.415311    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b869c385-b17d-4be3-b3ad-f5ba7e5eac48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b869c385-b17d-4be3-b3ad-f5ba7e5eac48" (UID: "b869c385-b17d-4be3-b3ad-f5ba7e5eac48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:20:03.516241 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:03.516214    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b869c385-b17d-4be3-b3ad-f5ba7e5eac48-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:20:04.218989 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:04.218955    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"
Jul 18 01:20:04.218989 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:04.218981    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk" event={"ID":"b869c385-b17d-4be3-b3ad-f5ba7e5eac48","Type":"ContainerDied","Data":"508756749a9887021e74e88687cffec7c471c9863b8cf9a7abc7cdd3de9e9d97"}
Jul 18 01:20:04.219236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:04.219034    2554 scope.go:117] "RemoveContainer" containerID="e8566454c72c8e8818a085a9cd9ceaef9c577b0204496d54d63ea0d6ca22fe59"
Jul 18 01:20:04.245874 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:04.245847    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"]
Jul 18 01:20:04.249207 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:04.249182    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ffmhk"]
Jul 18 01:20:04.337269 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:04.337234    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" path="/var/lib/kubelet/pods/b869c385-b17d-4be3-b3ad-f5ba7e5eac48/volumes"
Jul 18 01:20:18.763240 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763202    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"]
Jul 18 01:20:18.763671 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763600    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:18.763671 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763628    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:18.763671 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763642    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:18.763671 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763650    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:18.763671 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763661    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:18.763671 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763670    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763680    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763686    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763697    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763706    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763726    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763734    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763747    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763752    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763823    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763832    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763839    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763845    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763851    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:18.763902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.763858    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:18.768506 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.768485    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"
Jul 18 01:20:18.771637 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.771596    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:20:18.773877 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.773851    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"]
Jul 18 01:20:18.872426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.872388    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec40c7-43e8-4a0a-a0f9-a471863ef245-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2\" (UID: \"abec40c7-43e8-4a0a-a0f9-a471863ef245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"
Jul 18 01:20:18.973165 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.973129    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec40c7-43e8-4a0a-a0f9-a471863ef245-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2\" (UID: \"abec40c7-43e8-4a0a-a0f9-a471863ef245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"
Jul 18 01:20:18.973801 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:18.973782    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec40c7-43e8-4a0a-a0f9-a471863ef245-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2\" (UID: \"abec40c7-43e8-4a0a-a0f9-a471863ef245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"
Jul 18 01:20:19.081291 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:19.081214    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"
Jul 18 01:20:19.207704 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:19.207677    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"]
Jul 18 01:20:19.209802 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:20:19.209774    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabec40c7_43e8_4a0a_a0f9_a471863ef245.slice/crio-54b2d886363a0a0b2e5ad6bdd750e70c3e6bf87c1de5a436ee6194bba32104a7 WatchSource:0}: Error finding container 54b2d886363a0a0b2e5ad6bdd750e70c3e6bf87c1de5a436ee6194bba32104a7: Status 404 returned error can't find the container with id 54b2d886363a0a0b2e5ad6bdd750e70c3e6bf87c1de5a436ee6194bba32104a7
Jul 18 01:20:19.293566 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:19.293530    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" event={"ID":"abec40c7-43e8-4a0a-a0f9-a471863ef245","Type":"ContainerStarted","Data":"54b2d886363a0a0b2e5ad6bdd750e70c3e6bf87c1de5a436ee6194bba32104a7"}
Jul 18 01:20:20.300237 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:20.300201    2554 generic.go:356] "Generic (PLEG): container finished" podID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerID="927c54cbcb5bf19ade0e17499891a5868e5a125c40f9cf423dd54f459102f4fb" exitCode=6
Jul 18 01:20:20.300751 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:20.300289    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" event={"ID":"abec40c7-43e8-4a0a-a0f9-a471863ef245","Type":"ContainerDied","Data":"927c54cbcb5bf19ade0e17499891a5868e5a125c40f9cf423dd54f459102f4fb"}
Jul 18 01:20:20.300751 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:20.300474    2554 scope.go:117] "RemoveContainer" containerID="927c54cbcb5bf19ade0e17499891a5868e5a125c40f9cf423dd54f459102f4fb"
Jul 18 01:20:21.306931 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:21.306846    2554 generic.go:356] "Generic (PLEG): container finished" podID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerID="71be4296b70f373738398f942dc75e0d9feb5461a96c1690354a77070d0b448e" exitCode=6
Jul 18 01:20:21.307320 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:21.306932    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" event={"ID":"abec40c7-43e8-4a0a-a0f9-a471863ef245","Type":"ContainerDied","Data":"71be4296b70f373738398f942dc75e0d9feb5461a96c1690354a77070d0b448e"}
Jul 18 01:20:21.307320 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:21.306997    2554 scope.go:117] "RemoveContainer" containerID="927c54cbcb5bf19ade0e17499891a5868e5a125c40f9cf423dd54f459102f4fb"
Jul 18 01:20:21.307320 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:21.307254    2554 scope.go:117] "RemoveContainer" containerID="71be4296b70f373738398f942dc75e0d9feb5461a96c1690354a77070d0b448e"
Jul 18 01:20:21.307491 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:21.307465    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2_odh-ai-gateway-infra(abec40c7-43e8-4a0a-a0f9-a471863ef245)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245"
Jul 18 01:20:22.313870 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:22.313838    2554 scope.go:117] "RemoveContainer" containerID="71be4296b70f373738398f942dc75e0d9feb5461a96c1690354a77070d0b448e"
Jul 18 01:20:22.314271 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:22.314078    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2_odh-ai-gateway-infra(abec40c7-43e8-4a0a-a0f9-a471863ef245)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245"
Jul 18 01:20:31.912701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.912667    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"]
Jul 18 01:20:31.913100 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.913084    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:31.913100 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.913096    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:31.913179 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.913106    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:31.913179 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.913111    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:31.913179 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.913176    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ed9d96-c21e-47aa-81b8-d5961b2009cb" containerName="revoke-keys"
Jul 18 01:20:31.913275 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.913183    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b869c385-b17d-4be3-b3ad-f5ba7e5eac48" containerName="revoke-keys"
Jul 18 01:20:31.913275 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.913192    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4214e923-0269-4301-adff-3c8926d2ff45" containerName="revoke-keys"
Jul 18 01:20:31.917535 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.917513    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"
Jul 18 01:20:31.922780 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:31.922751    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"]
Jul 18 01:20:32.102552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:32.102516    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b028ef-fedd-4f41-beea-f29c2e4bc0c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s\" (UID: \"87b028ef-fedd-4f41-beea-f29c2e4bc0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"
Jul 18 01:20:32.203749 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:32.203674    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b028ef-fedd-4f41-beea-f29c2e4bc0c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s\" (UID: \"87b028ef-fedd-4f41-beea-f29c2e4bc0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"
Jul 18 01:20:32.204297 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:32.204276    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b028ef-fedd-4f41-beea-f29c2e4bc0c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s\" (UID: \"87b028ef-fedd-4f41-beea-f29c2e4bc0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"
Jul 18 01:20:32.230603 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:32.230571    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"
Jul 18 01:20:32.365354 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:32.365331    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"]
Jul 18 01:20:32.366933 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:20:32.366903    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b028ef_fedd_4f41_beea_f29c2e4bc0c8.slice/crio-e4fe99501a3213101ae8b4c485d65c18296dcad91bf67fc0c1f219079d900e0a WatchSource:0}: Error finding container e4fe99501a3213101ae8b4c485d65c18296dcad91bf67fc0c1f219079d900e0a: Status 404 returned error can't find the container with id e4fe99501a3213101ae8b4c485d65c18296dcad91bf67fc0c1f219079d900e0a
Jul 18 01:20:33.330796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:33.330764    2554 scope.go:117] "RemoveContainer" containerID="71be4296b70f373738398f942dc75e0d9feb5461a96c1690354a77070d0b448e"
Jul 18 01:20:33.369347 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:33.369315    2554 generic.go:356] "Generic (PLEG): container finished" podID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerID="135ca916f1125cc2b74b2eb4d6052dc764c727e25708bd92f0bb3fcfb13eb7d6" exitCode=7
Jul 18 01:20:33.369500 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:33.369398    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" event={"ID":"87b028ef-fedd-4f41-beea-f29c2e4bc0c8","Type":"ContainerDied","Data":"135ca916f1125cc2b74b2eb4d6052dc764c727e25708bd92f0bb3fcfb13eb7d6"}
Jul 18 01:20:33.369500 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:33.369469    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" event={"ID":"87b028ef-fedd-4f41-beea-f29c2e4bc0c8","Type":"ContainerStarted","Data":"e4fe99501a3213101ae8b4c485d65c18296dcad91bf67fc0c1f219079d900e0a"}
Jul 18 01:20:33.369584 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:33.369561    2554 scope.go:117] "RemoveContainer" containerID="135ca916f1125cc2b74b2eb4d6052dc764c727e25708bd92f0bb3fcfb13eb7d6"
Jul 18 01:20:34.375824 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.375790    2554 generic.go:356] "Generic (PLEG): container finished" podID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerID="9d338f74e2bf9b7e9fa1d8e72d1a20a748f6756bbd23c930f0d4a3abeb00c05d" exitCode=7
Jul 18 01:20:34.376266 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.375881    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" event={"ID":"87b028ef-fedd-4f41-beea-f29c2e4bc0c8","Type":"ContainerDied","Data":"9d338f74e2bf9b7e9fa1d8e72d1a20a748f6756bbd23c930f0d4a3abeb00c05d"}
Jul 18 01:20:34.376266 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.375943    2554 scope.go:117] "RemoveContainer" containerID="135ca916f1125cc2b74b2eb4d6052dc764c727e25708bd92f0bb3fcfb13eb7d6"
Jul 18 01:20:34.376266 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.376098    2554 scope.go:117] "RemoveContainer" containerID="9d338f74e2bf9b7e9fa1d8e72d1a20a748f6756bbd23c930f0d4a3abeb00c05d"
Jul 18 01:20:34.376445 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:34.376402    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s_odh-ai-gateway-infra(87b028ef-fedd-4f41-beea-f29c2e4bc0c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8"
Jul 18 01:20:34.377789 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.377771    2554 generic.go:356] "Generic (PLEG): container finished" podID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerID="7bc1768a470a8bcbcf762d72a89e099ec302311829c1122aad8afd83466062b5" exitCode=6
Jul 18 01:20:34.377905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.377820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" event={"ID":"abec40c7-43e8-4a0a-a0f9-a471863ef245","Type":"ContainerDied","Data":"7bc1768a470a8bcbcf762d72a89e099ec302311829c1122aad8afd83466062b5"}
Jul 18 01:20:34.378144 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.378129    2554 scope.go:117] "RemoveContainer" containerID="7bc1768a470a8bcbcf762d72a89e099ec302311829c1122aad8afd83466062b5"
Jul 18 01:20:34.378361 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:34.378342    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2_odh-ai-gateway-infra(abec40c7-43e8-4a0a-a0f9-a471863ef245)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245"
Jul 18 01:20:34.387054 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.387034    2554 scope.go:117] "RemoveContainer" containerID="71be4296b70f373738398f942dc75e0d9feb5461a96c1690354a77070d0b448e"
Jul 18 01:20:34.823094 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.823060    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"]
Jul 18 01:20:34.827766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.827744    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"
Jul 18 01:20:34.834801 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.834761    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"]
Jul 18 01:20:34.934235 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:34.934202    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec8a0b-b58e-4e0f-9204-6609c66a1fac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-x8ckn\" (UID: \"deec8a0b-b58e-4e0f-9204-6609c66a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"
Jul 18 01:20:35.035641 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.035586    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec8a0b-b58e-4e0f-9204-6609c66a1fac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-x8ckn\" (UID: \"deec8a0b-b58e-4e0f-9204-6609c66a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"
Jul 18 01:20:35.036198 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.036179    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec8a0b-b58e-4e0f-9204-6609c66a1fac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-x8ckn\" (UID: \"deec8a0b-b58e-4e0f-9204-6609c66a1fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"
Jul 18 01:20:35.139705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.139597    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"
Jul 18 01:20:35.285999 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.285974    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"]
Jul 18 01:20:35.288088 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:20:35.288060    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeec8a0b_b58e_4e0f_9204_6609c66a1fac.slice/crio-62c116b6c74c0325390c376b7f827b19023bd8ba9cad67556ddefe8cefb4bcc5 WatchSource:0}: Error finding container 62c116b6c74c0325390c376b7f827b19023bd8ba9cad67556ddefe8cefb4bcc5: Status 404 returned error can't find the container with id 62c116b6c74c0325390c376b7f827b19023bd8ba9cad67556ddefe8cefb4bcc5
Jul 18 01:20:35.384291 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.384256    2554 scope.go:117] "RemoveContainer" containerID="9d338f74e2bf9b7e9fa1d8e72d1a20a748f6756bbd23c930f0d4a3abeb00c05d"
Jul 18 01:20:35.384766 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:35.384522    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s_odh-ai-gateway-infra(87b028ef-fedd-4f41-beea-f29c2e4bc0c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8"
Jul 18 01:20:35.386796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.386773    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" event={"ID":"deec8a0b-b58e-4e0f-9204-6609c66a1fac","Type":"ContainerStarted","Data":"62c116b6c74c0325390c376b7f827b19023bd8ba9cad67556ddefe8cefb4bcc5"}
Jul 18 01:20:35.412924 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.412855    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"]
Jul 18 01:20:35.547449 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.547422    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"
Jul 18 01:20:35.642289 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.642256    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec40c7-43e8-4a0a-a0f9-a471863ef245-maas-api-service-ca\") pod \"abec40c7-43e8-4a0a-a0f9-a471863ef245\" (UID: \"abec40c7-43e8-4a0a-a0f9-a471863ef245\") "
Jul 18 01:20:35.642600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.642575    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abec40c7-43e8-4a0a-a0f9-a471863ef245-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abec40c7-43e8-4a0a-a0f9-a471863ef245" (UID: "abec40c7-43e8-4a0a-a0f9-a471863ef245"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:20:35.743836 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:35.743806    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec40c7-43e8-4a0a-a0f9-a471863ef245-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:20:36.393694 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.393659    2554 generic.go:356] "Generic (PLEG): container finished" podID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerID="0d8d61943010d262fa035d94bfa0255b30c1c1b035c518b54dc11a416ccb7f75" exitCode=6
Jul 18 01:20:36.394131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.393748    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" event={"ID":"deec8a0b-b58e-4e0f-9204-6609c66a1fac","Type":"ContainerDied","Data":"0d8d61943010d262fa035d94bfa0255b30c1c1b035c518b54dc11a416ccb7f75"}
Jul 18 01:20:36.394131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.393939    2554 scope.go:117] "RemoveContainer" containerID="0d8d61943010d262fa035d94bfa0255b30c1c1b035c518b54dc11a416ccb7f75"
Jul 18 01:20:36.395069 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.395053    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"
Jul 18 01:20:36.395157 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.395108    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2" event={"ID":"abec40c7-43e8-4a0a-a0f9-a471863ef245","Type":"ContainerDied","Data":"54b2d886363a0a0b2e5ad6bdd750e70c3e6bf87c1de5a436ee6194bba32104a7"}
Jul 18 01:20:36.395198 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.395159    2554 scope.go:117] "RemoveContainer" containerID="7bc1768a470a8bcbcf762d72a89e099ec302311829c1122aad8afd83466062b5"
Jul 18 01:20:36.422972 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.422945    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"]
Jul 18 01:20:36.428244 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.428221    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-j4xj2"]
Jul 18 01:20:36.809814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.809782    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"]
Jul 18 01:20:36.810205 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810193    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810206    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810215    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810221    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810227    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810233    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810293    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810304    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.810426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.810311    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" containerName="revoke-keys"
Jul 18 01:20:36.813408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.813391    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"
Jul 18 01:20:36.820627 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.820584    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"]
Jul 18 01:20:36.853317 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.853280    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6\" (UID: \"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"
Jul 18 01:20:36.954251 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.954207    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6\" (UID: \"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"
Jul 18 01:20:36.954867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:36.954848    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6\" (UID: \"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"
Jul 18 01:20:37.125928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:37.125898    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"
Jul 18 01:20:37.262045 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:37.262009    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"]
Jul 18 01:20:37.265039 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:20:37.265009    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384a277d_5fbd_44d1_bf7f_cf7ab0bb1ce2.slice/crio-b0f781a89c718b964c12f12d2f2bb244cda93bb42f715de8a9d56c97b4403585 WatchSource:0}: Error finding container b0f781a89c718b964c12f12d2f2bb244cda93bb42f715de8a9d56c97b4403585: Status 404 returned error can't find the container with id b0f781a89c718b964c12f12d2f2bb244cda93bb42f715de8a9d56c97b4403585
Jul 18 01:20:37.401914 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:37.401826    2554 generic.go:356] "Generic (PLEG): container finished" podID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerID="f704f1ac5e975845e96e74779faeefb304488efd3cdd2f30de3b32891c42d2b7" exitCode=6
Jul 18 01:20:37.402364 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:37.401913    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" event={"ID":"deec8a0b-b58e-4e0f-9204-6609c66a1fac","Type":"ContainerDied","Data":"f704f1ac5e975845e96e74779faeefb304488efd3cdd2f30de3b32891c42d2b7"}
Jul 18 01:20:37.402364 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:37.401972    2554 scope.go:117] "RemoveContainer" containerID="0d8d61943010d262fa035d94bfa0255b30c1c1b035c518b54dc11a416ccb7f75"
Jul 18 01:20:37.402364 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:37.402167    2554 scope.go:117] "RemoveContainer" containerID="f704f1ac5e975845e96e74779faeefb304488efd3cdd2f30de3b32891c42d2b7"
Jul 18 01:20:37.402532 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:37.402433    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-x8ckn_odh-ai-gateway-infra(deec8a0b-b58e-4e0f-9204-6609c66a1fac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac"
Jul 18 01:20:37.404045 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:37.404024    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" event={"ID":"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2","Type":"ContainerStarted","Data":"b0f781a89c718b964c12f12d2f2bb244cda93bb42f715de8a9d56c97b4403585"}
Jul 18 01:20:38.336701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:38.336669    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abec40c7-43e8-4a0a-a0f9-a471863ef245" path="/var/lib/kubelet/pods/abec40c7-43e8-4a0a-a0f9-a471863ef245/volumes"
Jul 18 01:20:38.417420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:38.417383    2554 scope.go:117] "RemoveContainer" containerID="f704f1ac5e975845e96e74779faeefb304488efd3cdd2f30de3b32891c42d2b7"
Jul 18 01:20:38.417879 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:38.417691    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-x8ckn_odh-ai-gateway-infra(deec8a0b-b58e-4e0f-9204-6609c66a1fac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac"
Jul 18 01:20:38.418806 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:38.418783    2554 generic.go:356] "Generic (PLEG): container finished" podID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerID="4fb61dd60a99ca2bc48cb7eff4cc66d6bd7e1f3ec64f3e5cd34909258dc0ccca" exitCode=6
Jul 18 01:20:38.418939 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:38.418863    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" event={"ID":"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2","Type":"ContainerDied","Data":"4fb61dd60a99ca2bc48cb7eff4cc66d6bd7e1f3ec64f3e5cd34909258dc0ccca"}
Jul 18 01:20:38.419040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:38.419023    2554 scope.go:117] "RemoveContainer" containerID="4fb61dd60a99ca2bc48cb7eff4cc66d6bd7e1f3ec64f3e5cd34909258dc0ccca"
Jul 18 01:20:39.425871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:39.425779    2554 generic.go:356] "Generic (PLEG): container finished" podID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerID="03d3760c860b2de0accd20ff650c921bd3abcab6b3cbe78a8aec480c0e7b77f4" exitCode=6
Jul 18 01:20:39.425871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:39.425816    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" event={"ID":"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2","Type":"ContainerDied","Data":"03d3760c860b2de0accd20ff650c921bd3abcab6b3cbe78a8aec480c0e7b77f4"}
Jul 18 01:20:39.425871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:39.425862    2554 scope.go:117] "RemoveContainer" containerID="4fb61dd60a99ca2bc48cb7eff4cc66d6bd7e1f3ec64f3e5cd34909258dc0ccca"
Jul 18 01:20:39.426373 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:39.426050    2554 scope.go:117] "RemoveContainer" containerID="03d3760c860b2de0accd20ff650c921bd3abcab6b3cbe78a8aec480c0e7b77f4"
Jul 18 01:20:39.426373 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:39.426300    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6_odh-ai-gateway-infra(384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2"
Jul 18 01:20:40.432517 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:40.432494    2554 scope.go:117] "RemoveContainer" containerID="03d3760c860b2de0accd20ff650c921bd3abcab6b3cbe78a8aec480c0e7b77f4"
Jul 18 01:20:40.432927 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:40.432729    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6_odh-ai-gateway-infra(384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2"
Jul 18 01:20:48.331010 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:48.330979    2554 scope.go:117] "RemoveContainer" containerID="9d338f74e2bf9b7e9fa1d8e72d1a20a748f6756bbd23c930f0d4a3abeb00c05d"
Jul 18 01:20:49.331197 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:49.331096    2554 scope.go:117] "RemoveContainer" containerID="f704f1ac5e975845e96e74779faeefb304488efd3cdd2f30de3b32891c42d2b7"
Jul 18 01:20:49.478906 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:49.478875    2554 generic.go:356] "Generic (PLEG): container finished" podID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerID="6aefb11e4e7e815eb9d471ef9304828701d21ee1296556aa833e818271ea2322" exitCode=7
Jul 18 01:20:49.479097 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:49.478950    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" event={"ID":"87b028ef-fedd-4f41-beea-f29c2e4bc0c8","Type":"ContainerDied","Data":"6aefb11e4e7e815eb9d471ef9304828701d21ee1296556aa833e818271ea2322"}
Jul 18 01:20:49.479097 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:49.479019    2554 scope.go:117] "RemoveContainer" containerID="9d338f74e2bf9b7e9fa1d8e72d1a20a748f6756bbd23c930f0d4a3abeb00c05d"
Jul 18 01:20:49.479402 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:49.479369    2554 scope.go:117] "RemoveContainer" containerID="6aefb11e4e7e815eb9d471ef9304828701d21ee1296556aa833e818271ea2322"
Jul 18 01:20:49.479666 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:49.479639    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s_odh-ai-gateway-infra(87b028ef-fedd-4f41-beea-f29c2e4bc0c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8"
Jul 18 01:20:50.485350 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.485315    2554 generic.go:356] "Generic (PLEG): container finished" podID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerID="e9aa51e538131c49be9b73a0d95120169e9a37d143b72a8104cb469d8c4836e3" exitCode=6
Jul 18 01:20:50.485818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.485388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" event={"ID":"deec8a0b-b58e-4e0f-9204-6609c66a1fac","Type":"ContainerDied","Data":"e9aa51e538131c49be9b73a0d95120169e9a37d143b72a8104cb469d8c4836e3"}
Jul 18 01:20:50.485818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.485443    2554 scope.go:117] "RemoveContainer" containerID="f704f1ac5e975845e96e74779faeefb304488efd3cdd2f30de3b32891c42d2b7"
Jul 18 01:20:50.485941 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.485830    2554 scope.go:117] "RemoveContainer" containerID="e9aa51e538131c49be9b73a0d95120169e9a37d143b72a8104cb469d8c4836e3"
Jul 18 01:20:50.486087 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:50.486065    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-x8ckn_odh-ai-gateway-infra(deec8a0b-b58e-4e0f-9204-6609c66a1fac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac"
Jul 18 01:20:50.506011 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.505983    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"]
Jul 18 01:20:50.640255 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.640226    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"
Jul 18 01:20:50.690927 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.690898    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b028ef-fedd-4f41-beea-f29c2e4bc0c8-maas-api-service-ca\") pod \"87b028ef-fedd-4f41-beea-f29c2e4bc0c8\" (UID: \"87b028ef-fedd-4f41-beea-f29c2e4bc0c8\") "
Jul 18 01:20:50.691283 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.691258    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b028ef-fedd-4f41-beea-f29c2e4bc0c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87b028ef-fedd-4f41-beea-f29c2e4bc0c8" (UID: "87b028ef-fedd-4f41-beea-f29c2e4bc0c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:20:50.792284 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:50.792249    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b028ef-fedd-4f41-beea-f29c2e4bc0c8-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:20:51.493490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.493455    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"
Jul 18 01:20:51.494109 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.493501    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s" event={"ID":"87b028ef-fedd-4f41-beea-f29c2e4bc0c8","Type":"ContainerDied","Data":"e4fe99501a3213101ae8b4c485d65c18296dcad91bf67fc0c1f219079d900e0a"}
Jul 18 01:20:51.494109 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.493559    2554 scope.go:117] "RemoveContainer" containerID="6aefb11e4e7e815eb9d471ef9304828701d21ee1296556aa833e818271ea2322"
Jul 18 01:20:51.511312 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.511287    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"]
Jul 18 01:20:51.518298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.518272    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"]
Jul 18 01:20:51.522903 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.522879    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-b957s"]
Jul 18 01:20:51.651330 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.651304    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"
Jul 18 01:20:51.703437 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.703404    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec8a0b-b58e-4e0f-9204-6609c66a1fac-maas-api-service-ca\") pod \"deec8a0b-b58e-4e0f-9204-6609c66a1fac\" (UID: \"deec8a0b-b58e-4e0f-9204-6609c66a1fac\") "
Jul 18 01:20:51.703808 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.703783    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deec8a0b-b58e-4e0f-9204-6609c66a1fac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "deec8a0b-b58e-4e0f-9204-6609c66a1fac" (UID: "deec8a0b-b58e-4e0f-9204-6609c66a1fac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:20:51.805067 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:51.804979    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec8a0b-b58e-4e0f-9204-6609c66a1fac-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:20:52.337464 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:52.337437    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" path="/var/lib/kubelet/pods/87b028ef-fedd-4f41-beea-f29c2e4bc0c8/volumes"
Jul 18 01:20:52.501916 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:52.501886    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"
Jul 18 01:20:52.501916 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:52.501899    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn" event={"ID":"deec8a0b-b58e-4e0f-9204-6609c66a1fac","Type":"ContainerDied","Data":"62c116b6c74c0325390c376b7f827b19023bd8ba9cad67556ddefe8cefb4bcc5"}
Jul 18 01:20:52.502468 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:52.501964    2554 scope.go:117] "RemoveContainer" containerID="e9aa51e538131c49be9b73a0d95120169e9a37d143b72a8104cb469d8c4836e3"
Jul 18 01:20:52.517252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:52.517226    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"]
Jul 18 01:20:52.525741 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:52.524648    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-x8ckn"]
Jul 18 01:20:54.336097 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:54.336064    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" path="/var/lib/kubelet/pods/deec8a0b-b58e-4e0f-9204-6609c66a1fac/volumes"
Jul 18 01:20:55.330569 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:55.330536    2554 scope.go:117] "RemoveContainer" containerID="03d3760c860b2de0accd20ff650c921bd3abcab6b3cbe78a8aec480c0e7b77f4"
Jul 18 01:20:56.526523 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:56.526491    2554 generic.go:356] "Generic (PLEG): container finished" podID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerID="b035daa37bb8cfda196b62174eb708acfb7afa124bfbb1cf509f863ead5323e8" exitCode=6
Jul 18 01:20:56.526991 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:56.526563    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" event={"ID":"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2","Type":"ContainerDied","Data":"b035daa37bb8cfda196b62174eb708acfb7afa124bfbb1cf509f863ead5323e8"}
Jul 18 01:20:56.526991 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:56.526639    2554 scope.go:117] "RemoveContainer" containerID="03d3760c860b2de0accd20ff650c921bd3abcab6b3cbe78a8aec480c0e7b77f4"
Jul 18 01:20:56.526991 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:56.526963    2554 scope.go:117] "RemoveContainer" containerID="b035daa37bb8cfda196b62174eb708acfb7afa124bfbb1cf509f863ead5323e8"
Jul 18 01:20:56.527207 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:20:56.527188    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6_odh-ai-gateway-infra(384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2"
Jul 18 01:20:57.552040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:57.552004    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"]
Jul 18 01:20:57.688866 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:57.688839    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"
Jul 18 01:20:57.767735 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:57.767699    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2-maas-api-service-ca\") pod \"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2\" (UID: \"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2\") "
Jul 18 01:20:57.768061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:57.768038    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" (UID: "384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:20:57.868902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:57.868819    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:20:58.539910 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:58.539879    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6" event={"ID":"384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2","Type":"ContainerDied","Data":"b0f781a89c718b964c12f12d2f2bb244cda93bb42f715de8a9d56c97b4403585"}
Jul 18 01:20:58.540103 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:58.539958    2554 scope.go:117] "RemoveContainer" containerID="b035daa37bb8cfda196b62174eb708acfb7afa124bfbb1cf509f863ead5323e8"
Jul 18 01:20:58.540103 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:58.539892    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"
Jul 18 01:20:58.556583 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:58.556558    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"]
Jul 18 01:20:58.563536 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:20:58.563509    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lr5x6"]
Jul 18 01:21:00.336183 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:00.336152    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" path="/var/lib/kubelet/pods/384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2/volumes"
Jul 18 01:21:08.803284 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803250    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"]
Jul 18 01:21:08.803718 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803704    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.803718 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803717    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803725    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803731    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803741    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803746    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803765    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803774    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803780    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803785    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803792    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.803795 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803798    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803805    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803810    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803817    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803823    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803888    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803897    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803904    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803912    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803921    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803930    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b028ef-fedd-4f41-beea-f29c2e4bc0c8" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803937    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.804091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.803942    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:08.808396 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.808379    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"
Jul 18 01:21:08.811082 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.811057    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:21:08.813268 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.813241    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"]
Jul 18 01:21:08.870923 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.870881    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c34812-38c6-4b36-904f-64a9d0ce1ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-cb46z\" (UID: \"05c34812-38c6-4b36-904f-64a9d0ce1ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"
Jul 18 01:21:08.971896 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.971857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c34812-38c6-4b36-904f-64a9d0ce1ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-cb46z\" (UID: \"05c34812-38c6-4b36-904f-64a9d0ce1ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"
Jul 18 01:21:08.972530 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:08.972504    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c34812-38c6-4b36-904f-64a9d0ce1ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-cb46z\" (UID: \"05c34812-38c6-4b36-904f-64a9d0ce1ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"
Jul 18 01:21:09.120896 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:09.120798    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"
Jul 18 01:21:09.460505 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:09.460480    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"]
Jul 18 01:21:09.462583 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:21:09.462552    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c34812_38c6_4b36_904f_64a9d0ce1ad8.slice/crio-b42d9e68c81d69ac3f39b1d66f843dc1048177828c2c18cb8558106b747e40a1 WatchSource:0}: Error finding container b42d9e68c81d69ac3f39b1d66f843dc1048177828c2c18cb8558106b747e40a1: Status 404 returned error can't find the container with id b42d9e68c81d69ac3f39b1d66f843dc1048177828c2c18cb8558106b747e40a1
Jul 18 01:21:09.597325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:09.597288    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" event={"ID":"05c34812-38c6-4b36-904f-64a9d0ce1ad8","Type":"ContainerStarted","Data":"b42d9e68c81d69ac3f39b1d66f843dc1048177828c2c18cb8558106b747e40a1"}
Jul 18 01:21:10.603898 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:10.603862    2554 generic.go:356] "Generic (PLEG): container finished" podID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerID="0e34c2a00cc7b9211d7b782bdbf793ac2fc3ecc9b0383527b2da7b5651be86af" exitCode=6
Jul 18 01:21:10.604305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:10.603948    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" event={"ID":"05c34812-38c6-4b36-904f-64a9d0ce1ad8","Type":"ContainerDied","Data":"0e34c2a00cc7b9211d7b782bdbf793ac2fc3ecc9b0383527b2da7b5651be86af"}
Jul 18 01:21:10.604305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:10.604172    2554 scope.go:117] "RemoveContainer" containerID="0e34c2a00cc7b9211d7b782bdbf793ac2fc3ecc9b0383527b2da7b5651be86af"
Jul 18 01:21:11.610772 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:11.610736    2554 generic.go:356] "Generic (PLEG): container finished" podID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerID="b1b2816c075daebdaa961947a7ca2846300665e0a826d62cfd4f92a1e4459f3f" exitCode=6
Jul 18 01:21:11.611216 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:11.610809    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" event={"ID":"05c34812-38c6-4b36-904f-64a9d0ce1ad8","Type":"ContainerDied","Data":"b1b2816c075daebdaa961947a7ca2846300665e0a826d62cfd4f92a1e4459f3f"}
Jul 18 01:21:11.611216 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:11.610891    2554 scope.go:117] "RemoveContainer" containerID="0e34c2a00cc7b9211d7b782bdbf793ac2fc3ecc9b0383527b2da7b5651be86af"
Jul 18 01:21:11.611216 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:11.610991    2554 scope.go:117] "RemoveContainer" containerID="b1b2816c075daebdaa961947a7ca2846300665e0a826d62cfd4f92a1e4459f3f"
Jul 18 01:21:11.611746 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:11.611231    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-cb46z_odh-ai-gateway-infra(05c34812-38c6-4b36-904f-64a9d0ce1ad8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8"
Jul 18 01:21:12.620106 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:12.620075    2554 scope.go:117] "RemoveContainer" containerID="b1b2816c075daebdaa961947a7ca2846300665e0a826d62cfd4f92a1e4459f3f"
Jul 18 01:21:12.620587 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:12.620368    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-cb46z_odh-ai-gateway-infra(05c34812-38c6-4b36-904f-64a9d0ce1ad8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8"
Jul 18 01:21:21.956386 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:21.956347    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"]
Jul 18 01:21:21.956839 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:21.956801    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:21.956839 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:21.956814    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="384a277d-5fbd-44d1-bf7f-cf7ab0bb1ce2" containerName="revoke-keys"
Jul 18 01:21:21.956918 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:21.956880    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="deec8a0b-b58e-4e0f-9204-6609c66a1fac" containerName="revoke-keys"
Jul 18 01:21:21.959975 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:21.959959    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"
Jul 18 01:21:21.966942 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:21.966916    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"]
Jul 18 01:21:22.096095 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:22.096055    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060c21cc-bd3e-4834-a8d9-5f9b613e6327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq\" (UID: \"060c21cc-bd3e-4834-a8d9-5f9b613e6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"
Jul 18 01:21:22.197129 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:22.197091    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060c21cc-bd3e-4834-a8d9-5f9b613e6327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq\" (UID: \"060c21cc-bd3e-4834-a8d9-5f9b613e6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"
Jul 18 01:21:22.197757 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:22.197736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060c21cc-bd3e-4834-a8d9-5f9b613e6327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq\" (UID: \"060c21cc-bd3e-4834-a8d9-5f9b613e6327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"
Jul 18 01:21:22.272534 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:22.272488    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"
Jul 18 01:21:22.406769 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:22.406743    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"]
Jul 18 01:21:22.408553 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:21:22.408523    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060c21cc_bd3e_4834_a8d9_5f9b613e6327.slice/crio-2cc85985cde8837c668deb271b4a65b5ce4b6dd99723d7dd8f0710caab4e343b WatchSource:0}: Error finding container 2cc85985cde8837c668deb271b4a65b5ce4b6dd99723d7dd8f0710caab4e343b: Status 404 returned error can't find the container with id 2cc85985cde8837c668deb271b4a65b5ce4b6dd99723d7dd8f0710caab4e343b
Jul 18 01:21:22.669095 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:22.669058    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" event={"ID":"060c21cc-bd3e-4834-a8d9-5f9b613e6327","Type":"ContainerStarted","Data":"2cc85985cde8837c668deb271b4a65b5ce4b6dd99723d7dd8f0710caab4e343b"}
Jul 18 01:21:23.330995 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:23.330966    2554 scope.go:117] "RemoveContainer" containerID="b1b2816c075daebdaa961947a7ca2846300665e0a826d62cfd4f92a1e4459f3f"
Jul 18 01:21:23.676275 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:23.676237    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" event={"ID":"05c34812-38c6-4b36-904f-64a9d0ce1ad8","Type":"ContainerStarted","Data":"abde255a0d3081b2679138168431a534f29d64da046db0bebaf7bd7677cdee18"}
Jul 18 01:21:23.677572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:23.677547    2554 generic.go:356] "Generic (PLEG): container finished" podID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerID="47daf246b29ac582b81ee9588aba2014db9fe016d82d3ed1528a27cddbdde574" exitCode=7
Jul 18 01:21:23.677702 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:23.677641    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" event={"ID":"060c21cc-bd3e-4834-a8d9-5f9b613e6327","Type":"ContainerDied","Data":"47daf246b29ac582b81ee9588aba2014db9fe016d82d3ed1528a27cddbdde574"}
Jul 18 01:21:23.677799 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:23.677785    2554 scope.go:117] "RemoveContainer" containerID="47daf246b29ac582b81ee9588aba2014db9fe016d82d3ed1528a27cddbdde574"
Jul 18 01:21:23.692125 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:23.692080    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" podStartSLOduration=15.692063386 podStartE2EDuration="15.692063386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:21:23.689917033 +0000 UTC m=+2439.815852012" watchObservedRunningTime="2026-07-18 01:21:23.692063386 +0000 UTC m=+2439.817998366"
Jul 18 01:21:24.684592 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.684499    2554 generic.go:356] "Generic (PLEG): container finished" podID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerID="abde255a0d3081b2679138168431a534f29d64da046db0bebaf7bd7677cdee18" exitCode=6
Jul 18 01:21:24.684592 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.684567    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" event={"ID":"05c34812-38c6-4b36-904f-64a9d0ce1ad8","Type":"ContainerDied","Data":"abde255a0d3081b2679138168431a534f29d64da046db0bebaf7bd7677cdee18"}
Jul 18 01:21:24.685140 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.684639    2554 scope.go:117] "RemoveContainer" containerID="b1b2816c075daebdaa961947a7ca2846300665e0a826d62cfd4f92a1e4459f3f"
Jul 18 01:21:24.685140 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.684994    2554 scope.go:117] "RemoveContainer" containerID="abde255a0d3081b2679138168431a534f29d64da046db0bebaf7bd7677cdee18"
Jul 18 01:21:24.685275 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:24.685253    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-cb46z_odh-ai-gateway-infra(05c34812-38c6-4b36-904f-64a9d0ce1ad8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8"
Jul 18 01:21:24.686502 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.686483    2554 generic.go:356] "Generic (PLEG): container finished" podID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerID="6899994a18a1e5649f9db411fa66dcf5f4f8fa8376509456926948573d95b452" exitCode=7
Jul 18 01:21:24.686607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.686530    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" event={"ID":"060c21cc-bd3e-4834-a8d9-5f9b613e6327","Type":"ContainerDied","Data":"6899994a18a1e5649f9db411fa66dcf5f4f8fa8376509456926948573d95b452"}
Jul 18 01:21:24.686777 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.686763    2554 scope.go:117] "RemoveContainer" containerID="6899994a18a1e5649f9db411fa66dcf5f4f8fa8376509456926948573d95b452"
Jul 18 01:21:24.686969 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:24.686951    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq_odh-ai-gateway-infra(060c21cc-bd3e-4834-a8d9-5f9b613e6327)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327"
Jul 18 01:21:24.695699 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.695680    2554 scope.go:117] "RemoveContainer" containerID="47daf246b29ac582b81ee9588aba2014db9fe016d82d3ed1528a27cddbdde574"
Jul 18 01:21:24.708604 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.708583    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"]
Jul 18 01:21:24.859426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.859389    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"]
Jul 18 01:21:24.863890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.863873    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"
Jul 18 01:21:24.869091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.869069    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"]
Jul 18 01:21:24.922933 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:24.922904    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaaec0b-2903-480e-9974-325896dd4154-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-qk94d\" (UID: \"1eaaec0b-2903-480e-9974-325896dd4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"
Jul 18 01:21:25.023597 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.023563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaaec0b-2903-480e-9974-325896dd4154-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-qk94d\" (UID: \"1eaaec0b-2903-480e-9974-325896dd4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"
Jul 18 01:21:25.024236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.024211    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaaec0b-2903-480e-9974-325896dd4154-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-qk94d\" (UID: \"1eaaec0b-2903-480e-9974-325896dd4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"
Jul 18 01:21:25.175779 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.175743    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"
Jul 18 01:21:25.308664 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.308640    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"]
Jul 18 01:21:25.310526 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:21:25.310491    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaaec0b_2903_480e_9974_325896dd4154.slice/crio-850f025496c9827bb7269bf6f7d7d3c58a751e6d128a0c42dba78f5d853a2fc9 WatchSource:0}: Error finding container 850f025496c9827bb7269bf6f7d7d3c58a751e6d128a0c42dba78f5d853a2fc9: Status 404 returned error can't find the container with id 850f025496c9827bb7269bf6f7d7d3c58a751e6d128a0c42dba78f5d853a2fc9
Jul 18 01:21:25.694464 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.694421    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" event={"ID":"1eaaec0b-2903-480e-9974-325896dd4154","Type":"ContainerStarted","Data":"cc66cf31e1cad06a06bd7576130fd588d5deeff67ec6539ada0a92eac931ec86"}
Jul 18 01:21:25.694886 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.694481    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" event={"ID":"1eaaec0b-2903-480e-9974-325896dd4154","Type":"ContainerStarted","Data":"850f025496c9827bb7269bf6f7d7d3c58a751e6d128a0c42dba78f5d853a2fc9"}
Jul 18 01:21:25.696118 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.696099    2554 scope.go:117] "RemoveContainer" containerID="6899994a18a1e5649f9db411fa66dcf5f4f8fa8376509456926948573d95b452"
Jul 18 01:21:25.696314 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:25.696292    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq_odh-ai-gateway-infra(060c21cc-bd3e-4834-a8d9-5f9b613e6327)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327"
Jul 18 01:21:25.724730 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.724652    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" podStartSLOduration=1.724627803 podStartE2EDuration="1.724627803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:21:25.722989971 +0000 UTC m=+2441.848924951" watchObservedRunningTime="2026-07-18 01:21:25.724627803 +0000 UTC m=+2441.850562783"
Jul 18 01:21:25.832940 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.832910    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"
Jul 18 01:21:25.932828 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.932739    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c34812-38c6-4b36-904f-64a9d0ce1ad8-maas-api-service-ca\") pod \"05c34812-38c6-4b36-904f-64a9d0ce1ad8\" (UID: \"05c34812-38c6-4b36-904f-64a9d0ce1ad8\") "
Jul 18 01:21:25.933115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:25.933091    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c34812-38c6-4b36-904f-64a9d0ce1ad8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05c34812-38c6-4b36-904f-64a9d0ce1ad8" (UID: "05c34812-38c6-4b36-904f-64a9d0ce1ad8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:21:26.036635 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.034425    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c34812-38c6-4b36-904f-64a9d0ce1ad8-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:26.702089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.702002    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"
Jul 18 01:21:26.702089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.702001    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z" event={"ID":"05c34812-38c6-4b36-904f-64a9d0ce1ad8","Type":"ContainerDied","Data":"b42d9e68c81d69ac3f39b1d66f843dc1048177828c2c18cb8558106b747e40a1"}
Jul 18 01:21:26.702605 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.702131    2554 scope.go:117] "RemoveContainer" containerID="abde255a0d3081b2679138168431a534f29d64da046db0bebaf7bd7677cdee18"
Jul 18 01:21:26.703656 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.703607    2554 generic.go:356] "Generic (PLEG): container finished" podID="1eaaec0b-2903-480e-9974-325896dd4154" containerID="cc66cf31e1cad06a06bd7576130fd588d5deeff67ec6539ada0a92eac931ec86" exitCode=6
Jul 18 01:21:26.703765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.703729    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" event={"ID":"1eaaec0b-2903-480e-9974-325896dd4154","Type":"ContainerDied","Data":"cc66cf31e1cad06a06bd7576130fd588d5deeff67ec6539ada0a92eac931ec86"}
Jul 18 01:21:26.703896 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.703874    2554 scope.go:117] "RemoveContainer" containerID="cc66cf31e1cad06a06bd7576130fd588d5deeff67ec6539ada0a92eac931ec86"
Jul 18 01:21:26.737487 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.737456    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"]
Jul 18 01:21:26.739278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:26.739245    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-cb46z"]
Jul 18 01:21:27.712837 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:27.712753    2554 generic.go:356] "Generic (PLEG): container finished" podID="1eaaec0b-2903-480e-9974-325896dd4154" containerID="6c55516b85e9d98921efb0876e9287bf52ac359474eda09eeb3a33da5c201a66" exitCode=6
Jul 18 01:21:27.712837 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:27.712826    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" event={"ID":"1eaaec0b-2903-480e-9974-325896dd4154","Type":"ContainerDied","Data":"6c55516b85e9d98921efb0876e9287bf52ac359474eda09eeb3a33da5c201a66"}
Jul 18 01:21:27.713325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:27.712865    2554 scope.go:117] "RemoveContainer" containerID="cc66cf31e1cad06a06bd7576130fd588d5deeff67ec6539ada0a92eac931ec86"
Jul 18 01:21:27.713325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:27.713133    2554 scope.go:117] "RemoveContainer" containerID="6c55516b85e9d98921efb0876e9287bf52ac359474eda09eeb3a33da5c201a66"
Jul 18 01:21:27.713433 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:27.713325    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-qk94d_odh-ai-gateway-infra(1eaaec0b-2903-480e-9974-325896dd4154)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" podUID="1eaaec0b-2903-480e-9974-325896dd4154"
Jul 18 01:21:28.337131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:28.337099    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" path="/var/lib/kubelet/pods/05c34812-38c6-4b36-904f-64a9d0ce1ad8/volumes"
Jul 18 01:21:28.719229 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:28.719153    2554 scope.go:117] "RemoveContainer" containerID="6c55516b85e9d98921efb0876e9287bf52ac359474eda09eeb3a33da5c201a66"
Jul 18 01:21:28.719596 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:28.719378    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-qk94d_odh-ai-gateway-infra(1eaaec0b-2903-480e-9974-325896dd4154)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" podUID="1eaaec0b-2903-480e-9974-325896dd4154"
Jul 18 01:21:36.859434 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859402    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"]
Jul 18 01:21:36.859883 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859869    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:21:36.859883 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859883    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:21:36.859962 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859893    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:21:36.859962 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859899    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:21:36.860032 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859975    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:21:36.860032 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859984    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:21:36.860032 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.859993    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:21:36.864399 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.864382    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"
Jul 18 01:21:36.869310 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:36.869277    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"]
Jul 18 01:21:37.048360 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.048323    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec33708f-592e-4fed-8acd-9d213211dcb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf\" (UID: \"ec33708f-592e-4fed-8acd-9d213211dcb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"
Jul 18 01:21:37.149787 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.149694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec33708f-592e-4fed-8acd-9d213211dcb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf\" (UID: \"ec33708f-592e-4fed-8acd-9d213211dcb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"
Jul 18 01:21:37.150408 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.150386    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec33708f-592e-4fed-8acd-9d213211dcb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf\" (UID: \"ec33708f-592e-4fed-8acd-9d213211dcb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"
Jul 18 01:21:37.177334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.177300    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"
Jul 18 01:21:37.313892 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.313862    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"]
Jul 18 01:21:37.316196 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:21:37.316169    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec33708f_592e_4fed_8acd_9d213211dcb7.slice/crio-42252e9a18363a7c8c1a257926dd02c0a131356eab7f04f08d332fd7f4626c57 WatchSource:0}: Error finding container 42252e9a18363a7c8c1a257926dd02c0a131356eab7f04f08d332fd7f4626c57: Status 404 returned error can't find the container with id 42252e9a18363a7c8c1a257926dd02c0a131356eab7f04f08d332fd7f4626c57
Jul 18 01:21:37.768006 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.767967    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" event={"ID":"ec33708f-592e-4fed-8acd-9d213211dcb7","Type":"ContainerStarted","Data":"9cdddc65686b0a7978825799bfe95deb80e24cc5e852ec39e701b333e90f72d0"}
Jul 18 01:21:37.768160 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.768052    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" event={"ID":"ec33708f-592e-4fed-8acd-9d213211dcb7","Type":"ContainerStarted","Data":"42252e9a18363a7c8c1a257926dd02c0a131356eab7f04f08d332fd7f4626c57"}
Jul 18 01:21:37.782808 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:37.782716    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" podStartSLOduration=1.782700685 podStartE2EDuration="1.782700685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:21:37.78213372 +0000 UTC m=+2453.908068701" watchObservedRunningTime="2026-07-18 01:21:37.782700685 +0000 UTC m=+2453.908635665"
Jul 18 01:21:38.774907 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:38.774870    2554 generic.go:356] "Generic (PLEG): container finished" podID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerID="9cdddc65686b0a7978825799bfe95deb80e24cc5e852ec39e701b333e90f72d0" exitCode=6
Jul 18 01:21:38.775370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:38.774943    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" event={"ID":"ec33708f-592e-4fed-8acd-9d213211dcb7","Type":"ContainerDied","Data":"9cdddc65686b0a7978825799bfe95deb80e24cc5e852ec39e701b333e90f72d0"}
Jul 18 01:21:38.775370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:38.775181    2554 scope.go:117] "RemoveContainer" containerID="9cdddc65686b0a7978825799bfe95deb80e24cc5e852ec39e701b333e90f72d0"
Jul 18 01:21:39.330861 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.330831    2554 scope.go:117] "RemoveContainer" containerID="6899994a18a1e5649f9db411fa66dcf5f4f8fa8376509456926948573d95b452"
Jul 18 01:21:39.331028 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.330863    2554 scope.go:117] "RemoveContainer" containerID="6c55516b85e9d98921efb0876e9287bf52ac359474eda09eeb3a33da5c201a66"
Jul 18 01:21:39.781985 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.781944    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" event={"ID":"1eaaec0b-2903-480e-9974-325896dd4154","Type":"ContainerStarted","Data":"97e2632a5d659f264e05c9396390b88d5269ce3fe732d212481255b31226d80a"}
Jul 18 01:21:39.783674 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.783650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" event={"ID":"060c21cc-bd3e-4834-a8d9-5f9b613e6327","Type":"ContainerStarted","Data":"32f2fe0ae119acc4b1df178bbb0a28149d8360639143e3db6376b0b0dabda2f2"}
Jul 18 01:21:39.785265 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.785241    2554 generic.go:356] "Generic (PLEG): container finished" podID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerID="f0db2da6ef36468419e5c43bfce4154067bded02d97d96ac686b7304a3c59d95" exitCode=6
Jul 18 01:21:39.785351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.785275    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" event={"ID":"ec33708f-592e-4fed-8acd-9d213211dcb7","Type":"ContainerDied","Data":"f0db2da6ef36468419e5c43bfce4154067bded02d97d96ac686b7304a3c59d95"}
Jul 18 01:21:39.785351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.785311    2554 scope.go:117] "RemoveContainer" containerID="9cdddc65686b0a7978825799bfe95deb80e24cc5e852ec39e701b333e90f72d0"
Jul 18 01:21:39.785519 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.785499    2554 scope.go:117] "RemoveContainer" containerID="f0db2da6ef36468419e5c43bfce4154067bded02d97d96ac686b7304a3c59d95"
Jul 18 01:21:39.785712 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:39.785694    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf_odh-ai-gateway-infra(ec33708f-592e-4fed-8acd-9d213211dcb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7"
Jul 18 01:21:39.828387 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:39.828338    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" podStartSLOduration=18.828322776 podStartE2EDuration="18.828322776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:21:39.826310214 +0000 UTC m=+2455.952245194" watchObservedRunningTime="2026-07-18 01:21:39.828322776 +0000 UTC m=+2455.954257755"
Jul 18 01:21:40.791529 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.791498    2554 generic.go:356] "Generic (PLEG): container finished" podID="1eaaec0b-2903-480e-9974-325896dd4154" containerID="97e2632a5d659f264e05c9396390b88d5269ce3fe732d212481255b31226d80a" exitCode=6
Jul 18 01:21:40.792001 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.791562    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" event={"ID":"1eaaec0b-2903-480e-9974-325896dd4154","Type":"ContainerDied","Data":"97e2632a5d659f264e05c9396390b88d5269ce3fe732d212481255b31226d80a"}
Jul 18 01:21:40.792001 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.791629    2554 scope.go:117] "RemoveContainer" containerID="6c55516b85e9d98921efb0876e9287bf52ac359474eda09eeb3a33da5c201a66"
Jul 18 01:21:40.792001 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.791917    2554 scope.go:117] "RemoveContainer" containerID="97e2632a5d659f264e05c9396390b88d5269ce3fe732d212481255b31226d80a"
Jul 18 01:21:40.792169 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:40.792152    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-qk94d_odh-ai-gateway-infra(1eaaec0b-2903-480e-9974-325896dd4154)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" podUID="1eaaec0b-2903-480e-9974-325896dd4154"
Jul 18 01:21:40.793694 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.793672    2554 generic.go:356] "Generic (PLEG): container finished" podID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerID="32f2fe0ae119acc4b1df178bbb0a28149d8360639143e3db6376b0b0dabda2f2" exitCode=7
Jul 18 01:21:40.793790 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.793741    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" event={"ID":"060c21cc-bd3e-4834-a8d9-5f9b613e6327","Type":"ContainerDied","Data":"32f2fe0ae119acc4b1df178bbb0a28149d8360639143e3db6376b0b0dabda2f2"}
Jul 18 01:21:40.794055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.794038    2554 scope.go:117] "RemoveContainer" containerID="32f2fe0ae119acc4b1df178bbb0a28149d8360639143e3db6376b0b0dabda2f2"
Jul 18 01:21:40.794264 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:40.794244    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq_odh-ai-gateway-infra(060c21cc-bd3e-4834-a8d9-5f9b613e6327)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327"
Jul 18 01:21:40.795315 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.795295    2554 scope.go:117] "RemoveContainer" containerID="f0db2da6ef36468419e5c43bfce4154067bded02d97d96ac686b7304a3c59d95"
Jul 18 01:21:40.795479 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:40.795460    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf_odh-ai-gateway-infra(ec33708f-592e-4fed-8acd-9d213211dcb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7"
Jul 18 01:21:40.802684 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.802666    2554 scope.go:117] "RemoveContainer" containerID="6899994a18a1e5649f9db411fa66dcf5f4f8fa8376509456926948573d95b452"
Jul 18 01:21:40.808071 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.808047    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"]
Jul 18 01:21:40.844673 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:40.844639    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"]
Jul 18 01:21:41.945735 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:41.945710    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"
Jul 18 01:21:41.957198 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:41.957171    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"]
Jul 18 01:21:41.957384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:41.957363    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy" containerID="cri-o://850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be" gracePeriod=30
Jul 18 01:21:41.977813 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:41.977787    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"
Jul 18 01:21:41.999518 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:41.999477    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060c21cc-bd3e-4834-a8d9-5f9b613e6327-maas-api-service-ca\") pod \"060c21cc-bd3e-4834-a8d9-5f9b613e6327\" (UID: \"060c21cc-bd3e-4834-a8d9-5f9b613e6327\") "
Jul 18 01:21:41.999705 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:41.999640    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaaec0b-2903-480e-9974-325896dd4154-maas-api-service-ca\") pod \"1eaaec0b-2903-480e-9974-325896dd4154\" (UID: \"1eaaec0b-2903-480e-9974-325896dd4154\") "
Jul 18 01:21:42.000026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:41.999891    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060c21cc-bd3e-4834-a8d9-5f9b613e6327-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "060c21cc-bd3e-4834-a8d9-5f9b613e6327" (UID: "060c21cc-bd3e-4834-a8d9-5f9b613e6327"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:21:42.000026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.000016    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eaaec0b-2903-480e-9974-325896dd4154-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1eaaec0b-2903-480e-9974-325896dd4154" (UID: "1eaaec0b-2903-480e-9974-325896dd4154"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:21:42.000185 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.000059    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/060c21cc-bd3e-4834-a8d9-5f9b613e6327-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:42.101374 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.101292    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaaec0b-2903-480e-9974-325896dd4154-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:42.809396 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.809367    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"
Jul 18 01:21:42.809596 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.809405    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d" event={"ID":"1eaaec0b-2903-480e-9974-325896dd4154","Type":"ContainerDied","Data":"850f025496c9827bb7269bf6f7d7d3c58a751e6d128a0c42dba78f5d853a2fc9"}
Jul 18 01:21:42.809596 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.809470    2554 scope.go:117] "RemoveContainer" containerID="97e2632a5d659f264e05c9396390b88d5269ce3fe732d212481255b31226d80a"
Jul 18 01:21:42.810691 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.810655    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"
Jul 18 01:21:42.810763 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.810651    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq" event={"ID":"060c21cc-bd3e-4834-a8d9-5f9b613e6327","Type":"ContainerDied","Data":"2cc85985cde8837c668deb271b4a65b5ce4b6dd99723d7dd8f0710caab4e343b"}
Jul 18 01:21:42.818152 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.818140    2554 scope.go:117] "RemoveContainer" containerID="32f2fe0ae119acc4b1df178bbb0a28149d8360639143e3db6376b0b0dabda2f2"
Jul 18 01:21:42.825575 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.825554    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"]
Jul 18 01:21:42.829155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.829134    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qk94d"]
Jul 18 01:21:42.843217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.843193    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"]
Jul 18 01:21:42.848349 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:42.848326    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5jpsq"]
Jul 18 01:21:44.335776 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:44.335746    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" path="/var/lib/kubelet/pods/060c21cc-bd3e-4834-a8d9-5f9b613e6327/volumes"
Jul 18 01:21:44.336150 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:44.336119    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eaaec0b-2903-480e-9974-325896dd4154" path="/var/lib/kubelet/pods/1eaaec0b-2903-480e-9974-325896dd4154/volumes"
Jul 18 01:21:44.536343 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:44.536306    2554 patch_prober.go:28] interesting pod/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 01:21:44.536502 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:44.536376    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 01:21:47.239388 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.239363    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:21:47.251678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251650    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-podinfo\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.251841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251692    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-certs\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.251841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251748    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-data\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.251841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251772    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-envoy\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.251841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251828    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-token\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.252055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251877    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq9st\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-kube-api-access-tq9st\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.252055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251974    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istiod-ca-cert\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.252055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.251979    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:21:47.252055 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252016    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-credential-socket\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.252252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252058    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-socket\") pod \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\" (UID: \"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868\") "
Jul 18 01:21:47.252252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252243    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:21:47.252454 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252424    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:21:47.252454 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252439    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.252578 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252460    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.252578 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252460    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:21:47.252692 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.252650    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-data" (OuterVolumeSpecName: "istio-data") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:21:47.254088 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.254057    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-kube-api-access-tq9st" (OuterVolumeSpecName: "kube-api-access-tq9st") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "kube-api-access-tq9st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:21:47.254217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.254198    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-token" (OuterVolumeSpecName: "istio-token") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:21:47.254389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.254363    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:21:47.254494 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.254367    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" (UID: "f94bc8b3-2e7b-4da1-bd09-df10eb1b2868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:21:47.353833 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.353797    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.353833 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.353827    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.353833 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.353835    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.354125 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.353844    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9st\" (UniqueName: \"kubernetes.io/projected/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-kube-api-access-tq9st\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.354125 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.353854    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.354125 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.353863    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.354125 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.353871    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.838863 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.838831    2554 generic.go:356] "Generic (PLEG): container finished" podID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerID="850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be" exitCode=0
Jul 18 01:21:47.839050 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.838883    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" event={"ID":"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868","Type":"ContainerDied","Data":"850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be"}
Jul 18 01:21:47.839050 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.838886    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"
Jul 18 01:21:47.839050 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.838927    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg" event={"ID":"f94bc8b3-2e7b-4da1-bd09-df10eb1b2868","Type":"ContainerDied","Data":"9ba92699fab6e03ee9804c0319a720365af17568174b4428541c65f4463e0ea5"}
Jul 18 01:21:47.839050 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.838952    2554 scope.go:117] "RemoveContainer" containerID="850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be"
Jul 18 01:21:47.848523 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.848501    2554 scope.go:117] "RemoveContainer" containerID="93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12"
Jul 18 01:21:47.856410 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.856392    2554 scope.go:117] "RemoveContainer" containerID="850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be"
Jul 18 01:21:47.856703 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:47.856684    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be\": container with ID starting with 850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be not found: ID does not exist" containerID="850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be"
Jul 18 01:21:47.856750 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.856712    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be"} err="failed to get container status \"850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be\": rpc error: code = NotFound desc = could not find container \"850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be\": container with ID starting with 850886e4cdfe6d6f64ae50fda81c82c98f4367dc721471df82623cb9cd4ed5be not found: ID does not exist"
Jul 18 01:21:47.856750 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.856730    2554 scope.go:117] "RemoveContainer" containerID="93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12"
Jul 18 01:21:47.856951 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:47.856933    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12\": container with ID starting with 93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12 not found: ID does not exist" containerID="93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12"
Jul 18 01:21:47.856995 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.856959    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12"} err="failed to get container status \"93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12\": rpc error: code = NotFound desc = could not find container \"93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12\": container with ID starting with 93523f8edacf32403362237fa618fa054b40c4514ac4b68cfc52c08c8816df12 not found: ID does not exist"
Jul 18 01:21:47.861015 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.860993    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"]
Jul 18 01:21:47.864585 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:47.864562    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4e26d7cc-openshift-default-6789cfdfc9-qh7cg"]
Jul 18 01:21:48.337838 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:48.337808    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" path="/var/lib/kubelet/pods/f94bc8b3-2e7b-4da1-bd09-df10eb1b2868/volumes"
Jul 18 01:21:53.330219 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:53.330188    2554 scope.go:117] "RemoveContainer" containerID="f0db2da6ef36468419e5c43bfce4154067bded02d97d96ac686b7304a3c59d95"
Jul 18 01:21:53.870824 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:53.870736    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" event={"ID":"ec33708f-592e-4fed-8acd-9d213211dcb7","Type":"ContainerStarted","Data":"f2ec15e6e72c3b009084c1539f6778c466009f21c1d664a330581156d903b9ce"}
Jul 18 01:21:54.877445 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:54.877413    2554 generic.go:356] "Generic (PLEG): container finished" podID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerID="f2ec15e6e72c3b009084c1539f6778c466009f21c1d664a330581156d903b9ce" exitCode=6
Jul 18 01:21:54.877445 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:54.877450    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" event={"ID":"ec33708f-592e-4fed-8acd-9d213211dcb7","Type":"ContainerDied","Data":"f2ec15e6e72c3b009084c1539f6778c466009f21c1d664a330581156d903b9ce"}
Jul 18 01:21:54.877963 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:54.877496    2554 scope.go:117] "RemoveContainer" containerID="f0db2da6ef36468419e5c43bfce4154067bded02d97d96ac686b7304a3c59d95"
Jul 18 01:21:54.877963 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:54.877833    2554 scope.go:117] "RemoveContainer" containerID="f2ec15e6e72c3b009084c1539f6778c466009f21c1d664a330581156d903b9ce"
Jul 18 01:21:54.878103 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:21:54.878082    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf_odh-ai-gateway-infra(ec33708f-592e-4fed-8acd-9d213211dcb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7"
Jul 18 01:21:54.899461 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:54.899434    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"]
Jul 18 01:21:56.020529 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.020501    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"
Jul 18 01:21:56.138852 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.138754    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec33708f-592e-4fed-8acd-9d213211dcb7-maas-api-service-ca\") pod \"ec33708f-592e-4fed-8acd-9d213211dcb7\" (UID: \"ec33708f-592e-4fed-8acd-9d213211dcb7\") "
Jul 18 01:21:56.139199 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.139168    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec33708f-592e-4fed-8acd-9d213211dcb7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec33708f-592e-4fed-8acd-9d213211dcb7" (UID: "ec33708f-592e-4fed-8acd-9d213211dcb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:21:56.239486 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.239452    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec33708f-592e-4fed-8acd-9d213211dcb7-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:21:56.890964 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.890923    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf" event={"ID":"ec33708f-592e-4fed-8acd-9d213211dcb7","Type":"ContainerDied","Data":"42252e9a18363a7c8c1a257926dd02c0a131356eab7f04f08d332fd7f4626c57"}
Jul 18 01:21:56.891161 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.890986    2554 scope.go:117] "RemoveContainer" containerID="f2ec15e6e72c3b009084c1539f6778c466009f21c1d664a330581156d903b9ce"
Jul 18 01:21:56.891161 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.890933    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"
Jul 18 01:21:56.905651 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.905610    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"]
Jul 18 01:21:56.909548 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:56.909521    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6cblf"]
Jul 18 01:21:58.336666 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:21:58.336628    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" path="/var/lib/kubelet/pods/ec33708f-592e-4fed-8acd-9d213211dcb7/volumes"
Jul 18 01:22:14.900987 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.900943    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"]
Jul 18 01:22:14.901514 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901494    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901518    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901531    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901539    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901555    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901564    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901576    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901584    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c34812-38c6-4b36-904f-64a9d0ce1ad8" containerName="revoke-keys"
Jul 18 01:22:14.901600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901596    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901605    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901655    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901665    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901680    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901687    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901700    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901709    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901717    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901726    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901741    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901749    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901845    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901858    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901867    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901876    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901885    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901893    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94bc8b3-2e7b-4da1-bd09-df10eb1b2868" containerName="istio-proxy"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901904    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901912    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="060c21cc-bd3e-4834-a8d9-5f9b613e6327" containerName="revoke-keys"
Jul 18 01:22:14.902029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.901921    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:14.906246 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.906224    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"
Jul 18 01:22:14.909290 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.909267    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:22:14.910799 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:14.910771    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"]
Jul 18 01:22:15.018551 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:15.018512    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7b19e0-dc3c-4e78-911f-691d8f30ccf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-nlhsl\" (UID: \"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"
Jul 18 01:22:15.120010 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:15.119973    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7b19e0-dc3c-4e78-911f-691d8f30ccf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-nlhsl\" (UID: \"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"
Jul 18 01:22:15.120669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:15.120611    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7b19e0-dc3c-4e78-911f-691d8f30ccf4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-nlhsl\" (UID: \"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"
Jul 18 01:22:15.218666 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:15.218541    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"
Jul 18 01:22:15.353897 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:15.353873    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"]
Jul 18 01:22:15.356388 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:22:15.356361    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7b19e0_dc3c_4e78_911f_691d8f30ccf4.slice/crio-6a26ed5a5c829e972e7d2ae16e88daa292b086677d6d5ac100a8871363cd25f1 WatchSource:0}: Error finding container 6a26ed5a5c829e972e7d2ae16e88daa292b086677d6d5ac100a8871363cd25f1: Status 404 returned error can't find the container with id 6a26ed5a5c829e972e7d2ae16e88daa292b086677d6d5ac100a8871363cd25f1
Jul 18 01:22:15.985303 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:15.985210    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" event={"ID":"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4","Type":"ContainerStarted","Data":"3b73e34bc5f30fec042de6f30d7802b4a5a5ec887db61bf4811a7bc7c56310ac"}
Jul 18 01:22:15.985303 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:15.985263    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" event={"ID":"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4","Type":"ContainerStarted","Data":"6a26ed5a5c829e972e7d2ae16e88daa292b086677d6d5ac100a8871363cd25f1"}
Jul 18 01:22:16.001400 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:16.001346    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" podStartSLOduration=2.001330515 podStartE2EDuration="2.001330515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:22:15.999098293 +0000 UTC m=+2492.125033272" watchObservedRunningTime="2026-07-18 01:22:16.001330515 +0000 UTC m=+2492.127265498"
Jul 18 01:22:16.990920 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:16.990881    2554 generic.go:356] "Generic (PLEG): container finished" podID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerID="3b73e34bc5f30fec042de6f30d7802b4a5a5ec887db61bf4811a7bc7c56310ac" exitCode=6
Jul 18 01:22:16.991411 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:16.990945    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" event={"ID":"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4","Type":"ContainerDied","Data":"3b73e34bc5f30fec042de6f30d7802b4a5a5ec887db61bf4811a7bc7c56310ac"}
Jul 18 01:22:16.991411 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:16.991270    2554 scope.go:117] "RemoveContainer" containerID="3b73e34bc5f30fec042de6f30d7802b4a5a5ec887db61bf4811a7bc7c56310ac"
Jul 18 01:22:17.997422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:17.997393    2554 generic.go:356] "Generic (PLEG): container finished" podID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerID="d524b69c802ac4545f389cb7856312579acfa0983d6e146301c261b50ee74474" exitCode=6
Jul 18 01:22:17.997873 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:17.997486    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" event={"ID":"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4","Type":"ContainerDied","Data":"d524b69c802ac4545f389cb7856312579acfa0983d6e146301c261b50ee74474"}
Jul 18 01:22:17.997873 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:17.997553    2554 scope.go:117] "RemoveContainer" containerID="3b73e34bc5f30fec042de6f30d7802b4a5a5ec887db61bf4811a7bc7c56310ac"
Jul 18 01:22:17.997873 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:17.997711    2554 scope.go:117] "RemoveContainer" containerID="d524b69c802ac4545f389cb7856312579acfa0983d6e146301c261b50ee74474"
Jul 18 01:22:17.998008 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:17.997937    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-nlhsl_odh-ai-gateway-infra(2a7b19e0-dc3c-4e78-911f-691d8f30ccf4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4"
Jul 18 01:22:19.005091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:19.005060    2554 scope.go:117] "RemoveContainer" containerID="d524b69c802ac4545f389cb7856312579acfa0983d6e146301c261b50ee74474"
Jul 18 01:22:19.005489 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:19.005254    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-nlhsl_odh-ai-gateway-infra(2a7b19e0-dc3c-4e78-911f-691d8f30ccf4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4"
Jul 18 01:22:22.003979 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.003938    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"]
Jul 18 01:22:22.004378 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.004324    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:22.004378 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.004335    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:22.004378 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.004363    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:22.004378 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.004371    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:22.004509 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.004430    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec33708f-592e-4fed-8acd-9d213211dcb7" containerName="revoke-keys"
Jul 18 01:22:22.004509 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.004442    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eaaec0b-2903-480e-9974-325896dd4154" containerName="revoke-keys"
Jul 18 01:22:22.007303 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.007287    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"
Jul 18 01:22:22.014426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.014397    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"]
Jul 18 01:22:22.086920 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.086885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6606af6c-4bad-42e8-b416-f6f33b07d46c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d\" (UID: \"6606af6c-4bad-42e8-b416-f6f33b07d46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"
Jul 18 01:22:22.187894 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.187853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6606af6c-4bad-42e8-b416-f6f33b07d46c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d\" (UID: \"6606af6c-4bad-42e8-b416-f6f33b07d46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"
Jul 18 01:22:22.188490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.188469    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6606af6c-4bad-42e8-b416-f6f33b07d46c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d\" (UID: \"6606af6c-4bad-42e8-b416-f6f33b07d46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"
Jul 18 01:22:22.319816 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.319725    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"
Jul 18 01:22:22.454841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:22.454816    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"]
Jul 18 01:22:22.457020 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:22:22.456993    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6606af6c_4bad_42e8_b416_f6f33b07d46c.slice/crio-09d8251bc9f62bc8c9d2b93785c8869e820879c97a77bf3dae446809376003e6 WatchSource:0}: Error finding container 09d8251bc9f62bc8c9d2b93785c8869e820879c97a77bf3dae446809376003e6: Status 404 returned error can't find the container with id 09d8251bc9f62bc8c9d2b93785c8869e820879c97a77bf3dae446809376003e6
Jul 18 01:22:23.034421 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:23.034384    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" event={"ID":"6606af6c-4bad-42e8-b416-f6f33b07d46c","Type":"ContainerStarted","Data":"f3620a08537c99fd2e7449e59d4e6dd42d79c849276b9be4cf81215dea04350f"}
Jul 18 01:22:23.034805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:23.034440    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" event={"ID":"6606af6c-4bad-42e8-b416-f6f33b07d46c","Type":"ContainerStarted","Data":"09d8251bc9f62bc8c9d2b93785c8869e820879c97a77bf3dae446809376003e6"}
Jul 18 01:22:23.050012 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:23.049954    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" podStartSLOduration=2.049938666 podStartE2EDuration="2.049938666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:22:23.048182024 +0000 UTC m=+2499.174117005" watchObservedRunningTime="2026-07-18 01:22:23.049938666 +0000 UTC m=+2499.175873645"
Jul 18 01:22:24.040121 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:24.040085    2554 generic.go:356] "Generic (PLEG): container finished" podID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerID="f3620a08537c99fd2e7449e59d4e6dd42d79c849276b9be4cf81215dea04350f" exitCode=6
Jul 18 01:22:24.040121 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:24.040125    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" event={"ID":"6606af6c-4bad-42e8-b416-f6f33b07d46c","Type":"ContainerDied","Data":"f3620a08537c99fd2e7449e59d4e6dd42d79c849276b9be4cf81215dea04350f"}
Jul 18 01:22:24.040707 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:24.040431    2554 scope.go:117] "RemoveContainer" containerID="f3620a08537c99fd2e7449e59d4e6dd42d79c849276b9be4cf81215dea04350f"
Jul 18 01:22:25.047338 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:25.047308    2554 generic.go:356] "Generic (PLEG): container finished" podID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerID="dc245091df84c1b0d4fa7562f6e40ef2f724eea0b54d7d464ca2e92d21460e9e" exitCode=6
Jul 18 01:22:25.047849 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:25.047388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" event={"ID":"6606af6c-4bad-42e8-b416-f6f33b07d46c","Type":"ContainerDied","Data":"dc245091df84c1b0d4fa7562f6e40ef2f724eea0b54d7d464ca2e92d21460e9e"}
Jul 18 01:22:25.047849 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:25.047465    2554 scope.go:117] "RemoveContainer" containerID="f3620a08537c99fd2e7449e59d4e6dd42d79c849276b9be4cf81215dea04350f"
Jul 18 01:22:25.047849 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:25.047682    2554 scope.go:117] "RemoveContainer" containerID="dc245091df84c1b0d4fa7562f6e40ef2f724eea0b54d7d464ca2e92d21460e9e"
Jul 18 01:22:25.048052 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:25.047934    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d_odh-ai-gateway-infra(6606af6c-4bad-42e8-b416-f6f33b07d46c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c"
Jul 18 01:22:26.055115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:26.055088    2554 scope.go:117] "RemoveContainer" containerID="dc245091df84c1b0d4fa7562f6e40ef2f724eea0b54d7d464ca2e92d21460e9e"
Jul 18 01:22:26.055517 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:26.055326    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d_odh-ai-gateway-infra(6606af6c-4bad-42e8-b416-f6f33b07d46c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c"
Jul 18 01:22:34.332915 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:34.332885    2554 scope.go:117] "RemoveContainer" containerID="d524b69c802ac4545f389cb7856312579acfa0983d6e146301c261b50ee74474"
Jul 18 01:22:35.101362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:35.101266    2554 generic.go:356] "Generic (PLEG): container finished" podID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerID="027d2b2c7f8e8a6648c417628884eda0bfd07ca5b5393dd191428876887cf64f" exitCode=6
Jul 18 01:22:35.101362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:35.101340    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" event={"ID":"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4","Type":"ContainerDied","Data":"027d2b2c7f8e8a6648c417628884eda0bfd07ca5b5393dd191428876887cf64f"}
Jul 18 01:22:35.101560 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:35.101407    2554 scope.go:117] "RemoveContainer" containerID="d524b69c802ac4545f389cb7856312579acfa0983d6e146301c261b50ee74474"
Jul 18 01:22:35.101798 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:35.101774    2554 scope.go:117] "RemoveContainer" containerID="027d2b2c7f8e8a6648c417628884eda0bfd07ca5b5393dd191428876887cf64f"
Jul 18 01:22:35.102037 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:35.102016    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-nlhsl_odh-ai-gateway-infra(2a7b19e0-dc3c-4e78-911f-691d8f30ccf4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4"
Jul 18 01:22:36.128589 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.128551    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"]
Jul 18 01:22:36.266665 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.266611    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"
Jul 18 01:22:36.323913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.323875    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7b19e0-dc3c-4e78-911f-691d8f30ccf4-maas-api-service-ca\") pod \"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4\" (UID: \"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4\") "
Jul 18 01:22:36.324227 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.324204    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7b19e0-dc3c-4e78-911f-691d8f30ccf4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" (UID: "2a7b19e0-dc3c-4e78-911f-691d8f30ccf4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:22:36.425504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.425423    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7b19e0-dc3c-4e78-911f-691d8f30ccf4-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:22:36.903782 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.903740    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"]
Jul 18 01:22:36.904152 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.904139    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:36.904196 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.904155    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:36.904196 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.904180    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:36.904196 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.904186    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:36.904295 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.904245    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:36.904295 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.904253    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:36.904295 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.904259    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:36.907323 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.907307    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"
Jul 18 01:22:36.913232 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:36.913209    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"]
Jul 18 01:22:37.031354 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.031320    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190378d-27d8-4157-b05f-124d7609ef62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x\" (UID: \"d190378d-27d8-4157-b05f-124d7609ef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"
Jul 18 01:22:37.114053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.114019    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"
Jul 18 01:22:37.114053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.114049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl" event={"ID":"2a7b19e0-dc3c-4e78-911f-691d8f30ccf4","Type":"ContainerDied","Data":"6a26ed5a5c829e972e7d2ae16e88daa292b086677d6d5ac100a8871363cd25f1"}
Jul 18 01:22:37.114274 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.114106    2554 scope.go:117] "RemoveContainer" containerID="027d2b2c7f8e8a6648c417628884eda0bfd07ca5b5393dd191428876887cf64f"
Jul 18 01:22:37.130168 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.130140    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"]
Jul 18 01:22:37.132000 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.131976    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-nlhsl"]
Jul 18 01:22:37.132107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.132052    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190378d-27d8-4157-b05f-124d7609ef62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x\" (UID: \"d190378d-27d8-4157-b05f-124d7609ef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"
Jul 18 01:22:37.132590 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.132575    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190378d-27d8-4157-b05f-124d7609ef62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x\" (UID: \"d190378d-27d8-4157-b05f-124d7609ef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"
Jul 18 01:22:37.218905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.218819    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"
Jul 18 01:22:37.355549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:37.355507    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"]
Jul 18 01:22:37.359719 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:22:37.359689    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd190378d_27d8_4157_b05f_124d7609ef62.slice/crio-fd4afbb9709ccba7800c5cf4467c6108f0f66b1261867d801aad159ddd92e9a9 WatchSource:0}: Error finding container fd4afbb9709ccba7800c5cf4467c6108f0f66b1261867d801aad159ddd92e9a9: Status 404 returned error can't find the container with id fd4afbb9709ccba7800c5cf4467c6108f0f66b1261867d801aad159ddd92e9a9
Jul 18 01:22:38.120947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:38.120916    2554 generic.go:356] "Generic (PLEG): container finished" podID="d190378d-27d8-4157-b05f-124d7609ef62" containerID="60af502da216d7bda4c4ab7436ede4e5c0d465f6cd74a153da96a93855002d90" exitCode=6
Jul 18 01:22:38.121090 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:38.120967    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" event={"ID":"d190378d-27d8-4157-b05f-124d7609ef62","Type":"ContainerDied","Data":"60af502da216d7bda4c4ab7436ede4e5c0d465f6cd74a153da96a93855002d90"}
Jul 18 01:22:38.121090 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:38.121025    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" event={"ID":"d190378d-27d8-4157-b05f-124d7609ef62","Type":"ContainerStarted","Data":"fd4afbb9709ccba7800c5cf4467c6108f0f66b1261867d801aad159ddd92e9a9"}
Jul 18 01:22:38.121213 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:38.121200    2554 scope.go:117] "RemoveContainer" containerID="60af502da216d7bda4c4ab7436ede4e5c0d465f6cd74a153da96a93855002d90"
Jul 18 01:22:38.336750 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:38.336707    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" path="/var/lib/kubelet/pods/2a7b19e0-dc3c-4e78-911f-691d8f30ccf4/volumes"
Jul 18 01:22:39.128823 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:39.128743    2554 generic.go:356] "Generic (PLEG): container finished" podID="d190378d-27d8-4157-b05f-124d7609ef62" containerID="a565f9d03e51b1694eb1e2ecf71ee8662f383244cc71eb9ecc83985b3f5b0ca6" exitCode=6
Jul 18 01:22:39.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:39.128831    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" event={"ID":"d190378d-27d8-4157-b05f-124d7609ef62","Type":"ContainerDied","Data":"a565f9d03e51b1694eb1e2ecf71ee8662f383244cc71eb9ecc83985b3f5b0ca6"}
Jul 18 01:22:39.128974 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:39.128880    2554 scope.go:117] "RemoveContainer" containerID="60af502da216d7bda4c4ab7436ede4e5c0d465f6cd74a153da96a93855002d90"
Jul 18 01:22:39.129138 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:39.129120    2554 scope.go:117] "RemoveContainer" containerID="a565f9d03e51b1694eb1e2ecf71ee8662f383244cc71eb9ecc83985b3f5b0ca6"
Jul 18 01:22:39.129385 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:39.129363    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x_odh-ai-gateway-infra(d190378d-27d8-4157-b05f-124d7609ef62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" podUID="d190378d-27d8-4157-b05f-124d7609ef62"
Jul 18 01:22:40.135201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:40.135175    2554 scope.go:117] "RemoveContainer" containerID="a565f9d03e51b1694eb1e2ecf71ee8662f383244cc71eb9ecc83985b3f5b0ca6"
Jul 18 01:22:40.135604 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:40.135383    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x_odh-ai-gateway-infra(d190378d-27d8-4157-b05f-124d7609ef62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" podUID="d190378d-27d8-4157-b05f-124d7609ef62"
Jul 18 01:22:40.331346 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:40.331313    2554 scope.go:117] "RemoveContainer" containerID="dc245091df84c1b0d4fa7562f6e40ef2f724eea0b54d7d464ca2e92d21460e9e"
Jul 18 01:22:41.141861 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:41.141826    2554 generic.go:356] "Generic (PLEG): container finished" podID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerID="0eb3421029e7cecabf2f69c087661c7321f3dcd74c191ecf707f87226e8c3429" exitCode=6
Jul 18 01:22:41.142299 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:41.141901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" event={"ID":"6606af6c-4bad-42e8-b416-f6f33b07d46c","Type":"ContainerDied","Data":"0eb3421029e7cecabf2f69c087661c7321f3dcd74c191ecf707f87226e8c3429"}
Jul 18 01:22:41.142299 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:41.141962    2554 scope.go:117] "RemoveContainer" containerID="dc245091df84c1b0d4fa7562f6e40ef2f724eea0b54d7d464ca2e92d21460e9e"
Jul 18 01:22:41.142299 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:41.142230    2554 scope.go:117] "RemoveContainer" containerID="0eb3421029e7cecabf2f69c087661c7321f3dcd74c191ecf707f87226e8c3429"
Jul 18 01:22:41.142488 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:41.142465    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d_odh-ai-gateway-infra(6606af6c-4bad-42e8-b416-f6f33b07d46c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c"
Jul 18 01:22:42.164787 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:42.164749    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"]
Jul 18 01:22:42.305025 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:42.304994    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"
Jul 18 01:22:42.384010 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:42.383981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6606af6c-4bad-42e8-b416-f6f33b07d46c-maas-api-service-ca\") pod \"6606af6c-4bad-42e8-b416-f6f33b07d46c\" (UID: \"6606af6c-4bad-42e8-b416-f6f33b07d46c\") "
Jul 18 01:22:42.384310 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:42.384289    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6606af6c-4bad-42e8-b416-f6f33b07d46c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6606af6c-4bad-42e8-b416-f6f33b07d46c" (UID: "6606af6c-4bad-42e8-b416-f6f33b07d46c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:22:42.485554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:42.485476    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6606af6c-4bad-42e8-b416-f6f33b07d46c-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:22:43.154996 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:43.154962    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"
Jul 18 01:22:43.154996 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:43.154974    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d" event={"ID":"6606af6c-4bad-42e8-b416-f6f33b07d46c","Type":"ContainerDied","Data":"09d8251bc9f62bc8c9d2b93785c8869e820879c97a77bf3dae446809376003e6"}
Jul 18 01:22:43.155235 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:43.155030    2554 scope.go:117] "RemoveContainer" containerID="0eb3421029e7cecabf2f69c087661c7321f3dcd74c191ecf707f87226e8c3429"
Jul 18 01:22:43.175536 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:43.175512    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"]
Jul 18 01:22:43.182275 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:43.182253    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-mdk5d"]
Jul 18 01:22:44.335823 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:44.335793    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" path="/var/lib/kubelet/pods/6606af6c-4bad-42e8-b416-f6f33b07d46c/volumes"
Jul 18 01:22:53.331146 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:53.331114    2554 scope.go:117] "RemoveContainer" containerID="a565f9d03e51b1694eb1e2ecf71ee8662f383244cc71eb9ecc83985b3f5b0ca6"
Jul 18 01:22:54.208351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:54.208267    2554 generic.go:356] "Generic (PLEG): container finished" podID="d190378d-27d8-4157-b05f-124d7609ef62" containerID="e8c070c256dc7ed0d9f59ce38cb9ac911f2f042387c9bc9b7235d8792b7512fa" exitCode=6
Jul 18 01:22:54.208535 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:54.208343    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" event={"ID":"d190378d-27d8-4157-b05f-124d7609ef62","Type":"ContainerDied","Data":"e8c070c256dc7ed0d9f59ce38cb9ac911f2f042387c9bc9b7235d8792b7512fa"}
Jul 18 01:22:54.208535 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:54.208410    2554 scope.go:117] "RemoveContainer" containerID="a565f9d03e51b1694eb1e2ecf71ee8662f383244cc71eb9ecc83985b3f5b0ca6"
Jul 18 01:22:54.208838 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:54.208821    2554 scope.go:117] "RemoveContainer" containerID="e8c070c256dc7ed0d9f59ce38cb9ac911f2f042387c9bc9b7235d8792b7512fa"
Jul 18 01:22:54.209075 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:22:54.209050    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x_odh-ai-gateway-infra(d190378d-27d8-4157-b05f-124d7609ef62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" podUID="d190378d-27d8-4157-b05f-124d7609ef62"
Jul 18 01:22:55.232251 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:55.232219    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"]
Jul 18 01:22:55.368673 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:55.368644    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"
Jul 18 01:22:55.515768 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:55.515735    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190378d-27d8-4157-b05f-124d7609ef62-maas-api-service-ca\") pod \"d190378d-27d8-4157-b05f-124d7609ef62\" (UID: \"d190378d-27d8-4157-b05f-124d7609ef62\") "
Jul 18 01:22:55.516125 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:55.516098    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d190378d-27d8-4157-b05f-124d7609ef62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d190378d-27d8-4157-b05f-124d7609ef62" (UID: "d190378d-27d8-4157-b05f-124d7609ef62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:22:55.616827 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:55.616791    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190378d-27d8-4157-b05f-124d7609ef62-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:22:56.221826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:56.221796    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"
Jul 18 01:22:56.221826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:56.221820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x" event={"ID":"d190378d-27d8-4157-b05f-124d7609ef62","Type":"ContainerDied","Data":"fd4afbb9709ccba7800c5cf4467c6108f0f66b1261867d801aad159ddd92e9a9"}
Jul 18 01:22:56.222193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:56.221878    2554 scope.go:117] "RemoveContainer" containerID="e8c070c256dc7ed0d9f59ce38cb9ac911f2f042387c9bc9b7235d8792b7512fa"
Jul 18 01:22:56.246013 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:56.245984    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"]
Jul 18 01:22:56.252505 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:56.252484    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-ftj9x"]
Jul 18 01:22:56.342872 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:56.342837    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d190378d-27d8-4157-b05f-124d7609ef62" path="/var/lib/kubelet/pods/d190378d-27d8-4157-b05f-124d7609ef62/volumes"
Jul 18 01:22:58.887091 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887056    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"]
Jul 18 01:22:58.887497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887446    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:22:58.887497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887458    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:22:58.887497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887469    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.887497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887475    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.887497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887491    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:58.887497 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887497    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7b19e0-dc3c-4e78-911f-691d8f30ccf4" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887503    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887508    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887517    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887522    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887536    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887542    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887602    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887625    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887632    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887638    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:22:58.887733 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.887644    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:22:58.892088 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.892069    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"
Jul 18 01:22:58.895203 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.895177    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:22:58.897175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.897139    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"]
Jul 18 01:22:58.949095 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:58.949061    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d46b9-eb18-4640-baa9-1dcccbb84d21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz\" (UID: \"ef5d46b9-eb18-4640-baa9-1dcccbb84d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"
Jul 18 01:22:59.050397 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:59.050360    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d46b9-eb18-4640-baa9-1dcccbb84d21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz\" (UID: \"ef5d46b9-eb18-4640-baa9-1dcccbb84d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"
Jul 18 01:22:59.051023 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:59.051001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d46b9-eb18-4640-baa9-1dcccbb84d21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz\" (UID: \"ef5d46b9-eb18-4640-baa9-1dcccbb84d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"
Jul 18 01:22:59.204731 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:59.204642    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"
Jul 18 01:22:59.340910 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:22:59.340884    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"]
Jul 18 01:22:59.343071 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:22:59.343047    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5d46b9_eb18_4640_baa9_1dcccbb84d21.slice/crio-497bb2fb52bd378d2fdab7496c1f9fd338855a27d064ed5bad37328151efdf78 WatchSource:0}: Error finding container 497bb2fb52bd378d2fdab7496c1f9fd338855a27d064ed5bad37328151efdf78: Status 404 returned error can't find the container with id 497bb2fb52bd378d2fdab7496c1f9fd338855a27d064ed5bad37328151efdf78
Jul 18 01:23:00.243602 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:00.243570    2554 generic.go:356] "Generic (PLEG): container finished" podID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerID="7070983d95b298d5ba794c8bb490a97fce700689cad844394c5a9c72a2ed6d4f" exitCode=6
Jul 18 01:23:00.244052 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:00.243642    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" event={"ID":"ef5d46b9-eb18-4640-baa9-1dcccbb84d21","Type":"ContainerDied","Data":"7070983d95b298d5ba794c8bb490a97fce700689cad844394c5a9c72a2ed6d4f"}
Jul 18 01:23:00.244052 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:00.243703    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" event={"ID":"ef5d46b9-eb18-4640-baa9-1dcccbb84d21","Type":"ContainerStarted","Data":"497bb2fb52bd378d2fdab7496c1f9fd338855a27d064ed5bad37328151efdf78"}
Jul 18 01:23:00.244052 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:00.243868    2554 scope.go:117] "RemoveContainer" containerID="7070983d95b298d5ba794c8bb490a97fce700689cad844394c5a9c72a2ed6d4f"
Jul 18 01:23:01.250797 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:01.250760    2554 generic.go:356] "Generic (PLEG): container finished" podID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerID="ab9f521fad51739fd34f3c21017cf85f82c596e2bf33bc2f9d1ba2ed49b91d76" exitCode=6
Jul 18 01:23:01.251234 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:01.250825    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" event={"ID":"ef5d46b9-eb18-4640-baa9-1dcccbb84d21","Type":"ContainerDied","Data":"ab9f521fad51739fd34f3c21017cf85f82c596e2bf33bc2f9d1ba2ed49b91d76"}
Jul 18 01:23:01.251234 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:01.250871    2554 scope.go:117] "RemoveContainer" containerID="7070983d95b298d5ba794c8bb490a97fce700689cad844394c5a9c72a2ed6d4f"
Jul 18 01:23:01.251234 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:01.251095    2554 scope.go:117] "RemoveContainer" containerID="ab9f521fad51739fd34f3c21017cf85f82c596e2bf33bc2f9d1ba2ed49b91d76"
Jul 18 01:23:01.251370 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:23:01.251315    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz_odh-ai-gateway-infra(ef5d46b9-eb18-4640-baa9-1dcccbb84d21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21"
Jul 18 01:23:02.256956 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:02.256929    2554 scope.go:117] "RemoveContainer" containerID="ab9f521fad51739fd34f3c21017cf85f82c596e2bf33bc2f9d1ba2ed49b91d76"
Jul 18 01:23:02.257356 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:23:02.257123    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz_odh-ai-gateway-infra(ef5d46b9-eb18-4640-baa9-1dcccbb84d21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21"
Jul 18 01:23:15.331003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:15.330974    2554 scope.go:117] "RemoveContainer" containerID="ab9f521fad51739fd34f3c21017cf85f82c596e2bf33bc2f9d1ba2ed49b91d76"
Jul 18 01:23:16.324259 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:16.324170    2554 generic.go:356] "Generic (PLEG): container finished" podID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerID="cbfc4a118a646604e2e2bd8341f9e5c2f420696da1adedba5b7cf091befe4db2" exitCode=6
Jul 18 01:23:16.324259 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:16.324218    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" event={"ID":"ef5d46b9-eb18-4640-baa9-1dcccbb84d21","Type":"ContainerDied","Data":"cbfc4a118a646604e2e2bd8341f9e5c2f420696da1adedba5b7cf091befe4db2"}
Jul 18 01:23:16.324514 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:16.324271    2554 scope.go:117] "RemoveContainer" containerID="ab9f521fad51739fd34f3c21017cf85f82c596e2bf33bc2f9d1ba2ed49b91d76"
Jul 18 01:23:16.324598 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:16.324577    2554 scope.go:117] "RemoveContainer" containerID="cbfc4a118a646604e2e2bd8341f9e5c2f420696da1adedba5b7cf091befe4db2"
Jul 18 01:23:16.324860 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:23:16.324834    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz_odh-ai-gateway-infra(ef5d46b9-eb18-4640-baa9-1dcccbb84d21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21"
Jul 18 01:23:17.366231 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:17.366194    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"]
Jul 18 01:23:17.520970 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:17.520940    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"
Jul 18 01:23:17.636898 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:17.636799    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d46b9-eb18-4640-baa9-1dcccbb84d21-maas-api-service-ca\") pod \"ef5d46b9-eb18-4640-baa9-1dcccbb84d21\" (UID: \"ef5d46b9-eb18-4640-baa9-1dcccbb84d21\") "
Jul 18 01:23:17.637227 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:17.637199    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5d46b9-eb18-4640-baa9-1dcccbb84d21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef5d46b9-eb18-4640-baa9-1dcccbb84d21" (UID: "ef5d46b9-eb18-4640-baa9-1dcccbb84d21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:23:17.737760 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:17.737721    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d46b9-eb18-4640-baa9-1dcccbb84d21-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:23:18.338626 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:18.338576    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz" event={"ID":"ef5d46b9-eb18-4640-baa9-1dcccbb84d21","Type":"ContainerDied","Data":"497bb2fb52bd378d2fdab7496c1f9fd338855a27d064ed5bad37328151efdf78"}
Jul 18 01:23:18.338626 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:18.338600    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"
Jul 18 01:23:18.338891 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:18.338658    2554 scope.go:117] "RemoveContainer" containerID="cbfc4a118a646604e2e2bd8341f9e5c2f420696da1adedba5b7cf091befe4db2"
Jul 18 01:23:18.362476 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:18.362448    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"]
Jul 18 01:23:18.366189 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:18.366163    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4jpmz"]
Jul 18 01:23:20.337107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:20.337074    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" path="/var/lib/kubelet/pods/ef5d46b9-eb18-4640-baa9-1dcccbb84d21/volumes"
Jul 18 01:23:28.940472 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940432    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"]
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940863    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940876    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940886    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940892    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940902    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940907    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940923    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940934    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6606af6c-4bad-42e8-b416-f6f33b07d46c" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.940994    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.941005    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:28.941137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.941018    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d190378d-27d8-4157-b05f-124d7609ef62" containerName="revoke-keys"
Jul 18 01:23:28.951751 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.951719    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:28.953721 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.953693    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"]
Jul 18 01:23:28.954233 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:28.954209    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d1e63f47-openshift-default-dockercfg-5bh8m\""
Jul 18 01:23:29.049994 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.049958    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2319f26-9438-4950-8d2b-0fd587e76876-istio-podinfo\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.049994 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.049997    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-socket\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.050334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.050014    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-certs\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.050334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.050045    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-envoy\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.050334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.050104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2319f26-9438-4950-8d2b-0fd587e76876-istiod-ca-cert\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.050334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.050138    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-istio-token\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.050334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.050175    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc65j\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-kube-api-access-cc65j\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.050334 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.050298    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-credential-socket\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.050732 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.050390    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-data\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151511 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151466    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-credential-socket\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151519    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-data\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151564    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2319f26-9438-4950-8d2b-0fd587e76876-istio-podinfo\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-socket\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151597    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-certs\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151649    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-envoy\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151673    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2319f26-9438-4950-8d2b-0fd587e76876-istiod-ca-cert\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.151739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151712    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-istio-token\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.152116 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc65j\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-kube-api-access-cc65j\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.152116 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151959    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-credential-socket\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.152116 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.151986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-data\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.152116 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.152030    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-socket\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.152288 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.152122    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-certs\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.152554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.152536    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2319f26-9438-4950-8d2b-0fd587e76876-istiod-ca-cert\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.154031 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.154004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2319f26-9438-4950-8d2b-0fd587e76876-istio-podinfo\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.154031 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.154019    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-envoy\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.159607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.159588    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-istio-token\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.159796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.159777    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc65j\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-kube-api-access-cc65j\") pod \"e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") " pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.262503 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.262476    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:29.397452 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.397429    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"]
Jul 18 01:23:29.399423 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:23:29.399399    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2319f26_9438_4950_8d2b_0fd587e76876.slice/crio-73bc1aaaed1de65724524f2276e262ebd4a5ad7e0e70d0f1ca8db8ba80931542 WatchSource:0}: Error finding container 73bc1aaaed1de65724524f2276e262ebd4a5ad7e0e70d0f1ca8db8ba80931542: Status 404 returned error can't find the container with id 73bc1aaaed1de65724524f2276e262ebd4a5ad7e0e70d0f1ca8db8ba80931542
Jul 18 01:23:29.401512 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.401468    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:23:29.401594 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.401561    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:23:29.401658 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:29.401611    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:23:30.399512 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:30.399468    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" event={"ID":"e2319f26-9438-4950-8d2b-0fd587e76876","Type":"ContainerStarted","Data":"ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c"}
Jul 18 01:23:30.400047 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:30.399538    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" event={"ID":"e2319f26-9438-4950-8d2b-0fd587e76876","Type":"ContainerStarted","Data":"73bc1aaaed1de65724524f2276e262ebd4a5ad7e0e70d0f1ca8db8ba80931542"}
Jul 18 01:23:30.424141 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:30.424093    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" podStartSLOduration=2.424077312 podStartE2EDuration="2.424077312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:23:30.422953304 +0000 UTC m=+2566.548888280" watchObservedRunningTime="2026-07-18 01:23:30.424077312 +0000 UTC m=+2566.550012290"
Jul 18 01:23:31.263283 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:31.263251    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:31.267934 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:31.267908    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:31.404459 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:31.404426    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:31.405441 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:31.405424    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:44.172945 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.172905    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"]
Jul 18 01:23:44.173678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.173657    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5d46b9-eb18-4640-baa9-1dcccbb84d21" containerName="revoke-keys"
Jul 18 01:23:44.177351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.177332    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"
Jul 18 01:23:44.180322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.180295    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:23:44.182969 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.182936    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"]
Jul 18 01:23:44.308116 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.308067    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ca9c53-5ef2-40cd-8053-0622ec37e16d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q\" (UID: \"c9ca9c53-5ef2-40cd-8053-0622ec37e16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"
Jul 18 01:23:44.408828 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.408784    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ca9c53-5ef2-40cd-8053-0622ec37e16d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q\" (UID: \"c9ca9c53-5ef2-40cd-8053-0622ec37e16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"
Jul 18 01:23:44.409437 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.409416    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ca9c53-5ef2-40cd-8053-0622ec37e16d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q\" (UID: \"c9ca9c53-5ef2-40cd-8053-0622ec37e16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"
Jul 18 01:23:44.493036 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.493005    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:23:44.500927 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.500903    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"
Jul 18 01:23:44.643660 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:44.643629    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"]
Jul 18 01:23:44.645203 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:23:44.645173    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ca9c53_5ef2_40cd_8053_0622ec37e16d.slice/crio-5c4692a32efb4d9d1833b3895189eb7be80f0b525818adffd607c804a1b75270 WatchSource:0}: Error finding container 5c4692a32efb4d9d1833b3895189eb7be80f0b525818adffd607c804a1b75270: Status 404 returned error can't find the container with id 5c4692a32efb4d9d1833b3895189eb7be80f0b525818adffd607c804a1b75270
Jul 18 01:23:45.477535 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:45.477496    2554 generic.go:356] "Generic (PLEG): container finished" podID="c9ca9c53-5ef2-40cd-8053-0622ec37e16d" containerID="9013dcbbb9b74e33189b062aa36f7c291f57419f4e2a8f4de4268ffd7d93ede1" exitCode=0
Jul 18 01:23:45.477953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:45.477578    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q" event={"ID":"c9ca9c53-5ef2-40cd-8053-0622ec37e16d","Type":"ContainerDied","Data":"9013dcbbb9b74e33189b062aa36f7c291f57419f4e2a8f4de4268ffd7d93ede1"}
Jul 18 01:23:45.477953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:45.477654    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q" event={"ID":"c9ca9c53-5ef2-40cd-8053-0622ec37e16d","Type":"ContainerStarted","Data":"5c4692a32efb4d9d1833b3895189eb7be80f0b525818adffd607c804a1b75270"}
Jul 18 01:23:46.405976 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.405935    2554 patch_prober.go:28] interesting pod/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Jul 18 01:23:46.406187 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.405990    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Jul 18 01:23:46.483746 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.483713    2554 generic.go:356] "Generic (PLEG): container finished" podID="e2319f26-9438-4950-8d2b-0fd587e76876" containerID="ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c" exitCode=0
Jul 18 01:23:46.484178 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.483787    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" event={"ID":"e2319f26-9438-4950-8d2b-0fd587e76876","Type":"ContainerDied","Data":"ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c"}
Jul 18 01:23:46.484748 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.484288    2554 scope.go:117] "RemoveContainer" containerID="ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c"
Jul 18 01:23:46.485306 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.485280    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:23:46.485370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.485345    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:23:46.485409 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.485374    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:23:46.646383 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.646355    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"
Jul 18 01:23:46.732363 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.732270    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ca9c53-5ef2-40cd-8053-0622ec37e16d-maas-api-service-ca\") pod \"c9ca9c53-5ef2-40cd-8053-0622ec37e16d\" (UID: \"c9ca9c53-5ef2-40cd-8053-0622ec37e16d\") "
Jul 18 01:23:46.732607 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.732581    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ca9c53-5ef2-40cd-8053-0622ec37e16d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9ca9c53-5ef2-40cd-8053-0622ec37e16d" (UID: "c9ca9c53-5ef2-40cd-8053-0622ec37e16d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:23:46.833254 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:46.833219    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ca9c53-5ef2-40cd-8053-0622ec37e16d-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:23:47.493805 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:47.493768    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q" event={"ID":"c9ca9c53-5ef2-40cd-8053-0622ec37e16d","Type":"ContainerDied","Data":"5c4692a32efb4d9d1833b3895189eb7be80f0b525818adffd607c804a1b75270"}
Jul 18 01:23:47.494245 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:47.493831    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4692a32efb4d9d1833b3895189eb7be80f0b525818adffd607c804a1b75270"
Jul 18 01:23:47.494245 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:47.493786    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q"
Jul 18 01:23:47.496104 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:47.496079    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" event={"ID":"e2319f26-9438-4950-8d2b-0fd587e76876","Type":"ContainerStarted","Data":"1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08"}
Jul 18 01:23:48.263247 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.263201    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:48.263598 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.263576    2554 patch_prober.go:28] interesting pod/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Jul 18 01:23:48.263679 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.263663    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Jul 18 01:23:48.934279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.934235    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"]
Jul 18 01:23:48.934882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.934860    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ca9c53-5ef2-40cd-8053-0622ec37e16d" containerName="revoke-keys"
Jul 18 01:23:48.934959 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.934884    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ca9c53-5ef2-40cd-8053-0622ec37e16d" containerName="revoke-keys"
Jul 18 01:23:48.935029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.935015    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ca9c53-5ef2-40cd-8053-0622ec37e16d" containerName="revoke-keys"
Jul 18 01:23:48.938652 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.938608    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"
Jul 18 01:23:48.941120 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.941101    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:23:48.945205 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:48.945186    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"]
Jul 18 01:23:49.057961 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.057924    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eebda8d-38b2-41a9-a859-9596f7a0ca6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx\" (UID: \"0eebda8d-38b2-41a9-a859-9596f7a0ca6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"
Jul 18 01:23:49.159660 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.159598    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eebda8d-38b2-41a9-a859-9596f7a0ca6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx\" (UID: \"0eebda8d-38b2-41a9-a859-9596f7a0ca6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"
Jul 18 01:23:49.160214 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.160193    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eebda8d-38b2-41a9-a859-9596f7a0ca6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx\" (UID: \"0eebda8d-38b2-41a9-a859-9596f7a0ca6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"
Jul 18 01:23:49.250391 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.250360    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"
Jul 18 01:23:49.263167 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.263132    2554 patch_prober.go:28] interesting pod/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Jul 18 01:23:49.263278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.263202    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Jul 18 01:23:49.401070 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.401046    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"]
Jul 18 01:23:49.403537 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:23:49.403504    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eebda8d_38b2_41a9_a859_9596f7a0ca6f.slice/crio-cf8fec865a2ecdcec03a5a87cb129053c8794b08164791897f43f29b8258828c WatchSource:0}: Error finding container cf8fec865a2ecdcec03a5a87cb129053c8794b08164791897f43f29b8258828c: Status 404 returned error can't find the container with id cf8fec865a2ecdcec03a5a87cb129053c8794b08164791897f43f29b8258828c
Jul 18 01:23:49.527147 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:49.527012    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" event={"ID":"0eebda8d-38b2-41a9-a859-9596f7a0ca6f","Type":"ContainerStarted","Data":"cf8fec865a2ecdcec03a5a87cb129053c8794b08164791897f43f29b8258828c"}
Jul 18 01:23:50.263106 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:50.263071    2554 patch_prober.go:28] interesting pod/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Jul 18 01:23:50.263513 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:50.263126    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Jul 18 01:23:50.535174 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:50.535082    2554 generic.go:356] "Generic (PLEG): container finished" podID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerID="9619a1a2cdca3d1016c2d623fbb2613da831016c729f8c1ea41bf2630073daa9" exitCode=6
Jul 18 01:23:50.535353 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:50.535168    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" event={"ID":"0eebda8d-38b2-41a9-a859-9596f7a0ca6f","Type":"ContainerDied","Data":"9619a1a2cdca3d1016c2d623fbb2613da831016c729f8c1ea41bf2630073daa9"}
Jul 18 01:23:50.535353 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:50.535345    2554 scope.go:117] "RemoveContainer" containerID="9619a1a2cdca3d1016c2d623fbb2613da831016c729f8c1ea41bf2630073daa9"
Jul 18 01:23:51.263872 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:51.263834    2554 patch_prober.go:28] interesting pod/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Jul 18 01:23:51.264361 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:51.264319    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Jul 18 01:23:51.542175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:51.542042    2554 generic.go:356] "Generic (PLEG): container finished" podID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerID="478fbe6877c76986921475fffd21a35363bc4efe51a2f0d34978d9735b94ea86" exitCode=6
Jul 18 01:23:51.542175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:51.542122    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" event={"ID":"0eebda8d-38b2-41a9-a859-9596f7a0ca6f","Type":"ContainerDied","Data":"478fbe6877c76986921475fffd21a35363bc4efe51a2f0d34978d9735b94ea86"}
Jul 18 01:23:51.542384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:51.542184    2554 scope.go:117] "RemoveContainer" containerID="9619a1a2cdca3d1016c2d623fbb2613da831016c729f8c1ea41bf2630073daa9"
Jul 18 01:23:51.542454 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:51.542402    2554 scope.go:117] "RemoveContainer" containerID="478fbe6877c76986921475fffd21a35363bc4efe51a2f0d34978d9735b94ea86"
Jul 18 01:23:51.542714 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:23:51.542691    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx_odh-ai-gateway-infra(0eebda8d-38b2-41a9-a859-9596f7a0ca6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f"
Jul 18 01:23:52.267059 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:52.267030    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:52.267456 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:52.267363    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:52.267980 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:52.267963    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:23:52.551545 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:23:52.551468    2554 scope.go:117] "RemoveContainer" containerID="478fbe6877c76986921475fffd21a35363bc4efe51a2f0d34978d9735b94ea86"
Jul 18 01:23:52.551744 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:23:52.551725    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx_odh-ai-gateway-infra(0eebda8d-38b2-41a9-a859-9596f7a0ca6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f"
Jul 18 01:24:03.330898 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:03.330865    2554 scope.go:117] "RemoveContainer" containerID="478fbe6877c76986921475fffd21a35363bc4efe51a2f0d34978d9735b94ea86"
Jul 18 01:24:03.609761 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:03.609673    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" event={"ID":"0eebda8d-38b2-41a9-a859-9596f7a0ca6f","Type":"ContainerStarted","Data":"2396ec0ece8e17aa46e12b287c6d24ad5deec3f0e2f9110387b12ce7cce6ce06"}
Jul 18 01:24:03.624508 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:03.624457    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" podStartSLOduration=15.624445549 podStartE2EDuration="15.624445549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:03.623305413 +0000 UTC m=+2599.749240392" watchObservedRunningTime="2026-07-18 01:24:03.624445549 +0000 UTC m=+2599.750380529"
Jul 18 01:24:04.616333 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:04.616298    2554 generic.go:356] "Generic (PLEG): container finished" podID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerID="2396ec0ece8e17aa46e12b287c6d24ad5deec3f0e2f9110387b12ce7cce6ce06" exitCode=6
Jul 18 01:24:04.616758 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:04.616371    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" event={"ID":"0eebda8d-38b2-41a9-a859-9596f7a0ca6f","Type":"ContainerDied","Data":"2396ec0ece8e17aa46e12b287c6d24ad5deec3f0e2f9110387b12ce7cce6ce06"}
Jul 18 01:24:04.616758 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:04.616434    2554 scope.go:117] "RemoveContainer" containerID="478fbe6877c76986921475fffd21a35363bc4efe51a2f0d34978d9735b94ea86"
Jul 18 01:24:04.616758 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:04.616753    2554 scope.go:117] "RemoveContainer" containerID="2396ec0ece8e17aa46e12b287c6d24ad5deec3f0e2f9110387b12ce7cce6ce06"
Jul 18 01:24:04.617021 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:04.616997    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx_odh-ai-gateway-infra(0eebda8d-38b2-41a9-a859-9596f7a0ca6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f"
Jul 18 01:24:04.633308 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:04.633279    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"]
Jul 18 01:24:05.759381 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:05.759353    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"
Jul 18 01:24:05.830297 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:05.830265    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eebda8d-38b2-41a9-a859-9596f7a0ca6f-maas-api-service-ca\") pod \"0eebda8d-38b2-41a9-a859-9596f7a0ca6f\" (UID: \"0eebda8d-38b2-41a9-a859-9596f7a0ca6f\") "
Jul 18 01:24:05.830700 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:05.830676    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eebda8d-38b2-41a9-a859-9596f7a0ca6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0eebda8d-38b2-41a9-a859-9596f7a0ca6f" (UID: "0eebda8d-38b2-41a9-a859-9596f7a0ca6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:24:05.932031 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:05.931946    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eebda8d-38b2-41a9-a859-9596f7a0ca6f-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:06.629744 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:06.629663    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"
Jul 18 01:24:06.629744 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:06.629684    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx" event={"ID":"0eebda8d-38b2-41a9-a859-9596f7a0ca6f","Type":"ContainerDied","Data":"cf8fec865a2ecdcec03a5a87cb129053c8794b08164791897f43f29b8258828c"}
Jul 18 01:24:06.629980 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:06.629771    2554 scope.go:117] "RemoveContainer" containerID="2396ec0ece8e17aa46e12b287c6d24ad5deec3f0e2f9110387b12ce7cce6ce06"
Jul 18 01:24:06.646698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:06.646673    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"]
Jul 18 01:24:06.652211 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:06.652187    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-dk5jx"]
Jul 18 01:24:08.336691 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:08.336661    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" path="/var/lib/kubelet/pods/0eebda8d-38b2-41a9-a859-9596f7a0ca6f/volumes"
Jul 18 01:24:09.931319 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:09.931283    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"]
Jul 18 01:24:09.931762 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:09.931545    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy" containerID="cri-o://1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08" gracePeriod=30
Jul 18 01:24:15.190216 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.190191    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:24:15.220456 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220382    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-socket\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.220456 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220416    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-data\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.220456 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220450    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-certs\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.220765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220510    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2319f26-9438-4950-8d2b-0fd587e76876-istio-podinfo\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.220765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220555    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2319f26-9438-4950-8d2b-0fd587e76876-istiod-ca-cert\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.220765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220586    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-envoy\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.220765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220666    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:24:15.220765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220676    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc65j\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-kube-api-access-cc65j\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.220765 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220762    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-istio-token\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.221094 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220790    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:24:15.221094 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.220801    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-credential-socket\") pod \"e2319f26-9438-4950-8d2b-0fd587e76876\" (UID: \"e2319f26-9438-4950-8d2b-0fd587e76876\") "
Jul 18 01:24:15.221230 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.221197    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.221230 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.221222    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2319f26-9438-4950-8d2b-0fd587e76876-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:24:15.221346 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.221236    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.221406 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.221342    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:24:15.221467 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.221444    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:24:15.222790 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.222762    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2319f26-9438-4950-8d2b-0fd587e76876-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:24:15.223130 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.223103    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:24:15.223231 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.223194    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-kube-api-access-cc65j" (OuterVolumeSpecName: "kube-api-access-cc65j") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "kube-api-access-cc65j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:24:15.223231 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.223213    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2319f26-9438-4950-8d2b-0fd587e76876" (UID: "e2319f26-9438-4950-8d2b-0fd587e76876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:24:15.322416 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.322382    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc65j\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-kube-api-access-cc65j\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.322416 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.322411    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2319f26-9438-4950-8d2b-0fd587e76876-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.322416 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.322423    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.322698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.322458    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.322698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.322470    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2319f26-9438-4950-8d2b-0fd587e76876-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.322698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.322483    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2319f26-9438-4950-8d2b-0fd587e76876-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.322698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.322494    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2319f26-9438-4950-8d2b-0fd587e76876-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:15.674182 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.674148    2554 generic.go:356] "Generic (PLEG): container finished" podID="e2319f26-9438-4950-8d2b-0fd587e76876" containerID="1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08" exitCode=0
Jul 18 01:24:15.674376 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.674191    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" event={"ID":"e2319f26-9438-4950-8d2b-0fd587e76876","Type":"ContainerDied","Data":"1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08"}
Jul 18 01:24:15.674376 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.674202    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"
Jul 18 01:24:15.674376 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.674249    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4" event={"ID":"e2319f26-9438-4950-8d2b-0fd587e76876","Type":"ContainerDied","Data":"73bc1aaaed1de65724524f2276e262ebd4a5ad7e0e70d0f1ca8db8ba80931542"}
Jul 18 01:24:15.674376 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.674280    2554 scope.go:117] "RemoveContainer" containerID="1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08"
Jul 18 01:24:15.683512 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.683488    2554 scope.go:117] "RemoveContainer" containerID="ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c"
Jul 18 01:24:15.695222 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.695202    2554 scope.go:117] "RemoveContainer" containerID="1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08"
Jul 18 01:24:15.695507 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:15.695488    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08\": container with ID starting with 1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08 not found: ID does not exist" containerID="1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08"
Jul 18 01:24:15.695562 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.695516    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08"} err="failed to get container status \"1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08\": rpc error: code = NotFound desc = could not find container \"1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08\": container with ID starting with 1d0293a177a71d54871fdccedbd4a2af02d87a1866b89e8dbaddca3cfc9f6a08 not found: ID does not exist"
Jul 18 01:24:15.695562 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.695538    2554 scope.go:117] "RemoveContainer" containerID="ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c"
Jul 18 01:24:15.695672 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.695594    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"]
Jul 18 01:24:15.695803 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:15.695783    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c\": container with ID starting with ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c not found: ID does not exist" containerID="ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c"
Jul 18 01:24:15.695848 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.695808    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c"} err="failed to get container status \"ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c\": rpc error: code = NotFound desc = could not find container \"ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c\": container with ID starting with ac219f6be13b8d52d9b0391942ab24e892cf05311b69eebb3206e633f112740c not found: ID does not exist"
Jul 18 01:24:15.699591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:15.699569    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1e63f47-openshift-default-55df57c495-2x4c4"]
Jul 18 01:24:16.336696 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:16.336665    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" path="/var/lib/kubelet/pods/e2319f26-9438-4950-8d2b-0fd587e76876/volumes"
Jul 18 01:24:17.020963 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.020929    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"]
Jul 18 01:24:17.021343 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021330    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:17.021389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021344    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:17.021389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021364    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:17.021389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021370    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:17.021389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021380    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy"
Jul 18 01:24:17.021389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021385    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy"
Jul 18 01:24:17.021865 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021391    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy"
Jul 18 01:24:17.021865 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021396    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy"
Jul 18 01:24:17.021865 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021451    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy"
Jul 18 01:24:17.021865 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021460    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:17.021865 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021465    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:17.021865 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.021472    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:17.025861 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.025843    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"
Jul 18 01:24:17.029711 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.029692    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:24:17.041296 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.041274    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"]
Jul 18 01:24:17.141383 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.141348    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713d76-78c6-4eae-8aca-9dd5b4cff7e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t\" (UID: \"52713d76-78c6-4eae-8aca-9dd5b4cff7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"
Jul 18 01:24:17.242823 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.242778    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713d76-78c6-4eae-8aca-9dd5b4cff7e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t\" (UID: \"52713d76-78c6-4eae-8aca-9dd5b4cff7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"
Jul 18 01:24:17.243443 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.243422    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713d76-78c6-4eae-8aca-9dd5b4cff7e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t\" (UID: \"52713d76-78c6-4eae-8aca-9dd5b4cff7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"
Jul 18 01:24:17.336397 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.336320    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"
Jul 18 01:24:17.469829 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.469799    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"]
Jul 18 01:24:17.471793 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:24:17.471761    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52713d76_78c6_4eae_8aca_9dd5b4cff7e3.slice/crio-ef2bc0c98d36b67ba30ea225c894979c01660165ee6ee8071ac7d7ea439e8fdc WatchSource:0}: Error finding container ef2bc0c98d36b67ba30ea225c894979c01660165ee6ee8071ac7d7ea439e8fdc: Status 404 returned error can't find the container with id ef2bc0c98d36b67ba30ea225c894979c01660165ee6ee8071ac7d7ea439e8fdc
Jul 18 01:24:17.687933 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:17.687850    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" event={"ID":"52713d76-78c6-4eae-8aca-9dd5b4cff7e3","Type":"ContainerStarted","Data":"ef2bc0c98d36b67ba30ea225c894979c01660165ee6ee8071ac7d7ea439e8fdc"}
Jul 18 01:24:18.693567 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:18.693533    2554 generic.go:356] "Generic (PLEG): container finished" podID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerID="b3456ac2934f69deb9b176cbaacfbad9aafb46f2bff68c6ca790e67288be260a" exitCode=6
Jul 18 01:24:18.694016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:18.693639    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" event={"ID":"52713d76-78c6-4eae-8aca-9dd5b4cff7e3","Type":"ContainerDied","Data":"b3456ac2934f69deb9b176cbaacfbad9aafb46f2bff68c6ca790e67288be260a"}
Jul 18 01:24:18.694016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:18.693768    2554 scope.go:117] "RemoveContainer" containerID="b3456ac2934f69deb9b176cbaacfbad9aafb46f2bff68c6ca790e67288be260a"
Jul 18 01:24:19.701184 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:19.701090    2554 generic.go:356] "Generic (PLEG): container finished" podID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerID="5bcc770dcb511814d21c9f1cfe1f13816d56b9e6aaa3fde11c244e40dd11e293" exitCode=6
Jul 18 01:24:19.701586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:19.701183    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" event={"ID":"52713d76-78c6-4eae-8aca-9dd5b4cff7e3","Type":"ContainerDied","Data":"5bcc770dcb511814d21c9f1cfe1f13816d56b9e6aaa3fde11c244e40dd11e293"}
Jul 18 01:24:19.701586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:19.701245    2554 scope.go:117] "RemoveContainer" containerID="b3456ac2934f69deb9b176cbaacfbad9aafb46f2bff68c6ca790e67288be260a"
Jul 18 01:24:19.701586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:19.701386    2554 scope.go:117] "RemoveContainer" containerID="5bcc770dcb511814d21c9f1cfe1f13816d56b9e6aaa3fde11c244e40dd11e293"
Jul 18 01:24:19.701735 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:19.701644    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t_odh-ai-gateway-infra(52713d76-78c6-4eae-8aca-9dd5b4cff7e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3"
Jul 18 01:24:20.707269 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:20.707243    2554 scope.go:117] "RemoveContainer" containerID="5bcc770dcb511814d21c9f1cfe1f13816d56b9e6aaa3fde11c244e40dd11e293"
Jul 18 01:24:20.707679 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:20.707424    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t_odh-ai-gateway-infra(52713d76-78c6-4eae-8aca-9dd5b4cff7e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3"
Jul 18 01:24:29.601810 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.601768    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"]
Jul 18 01:24:29.602403 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.602383    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:29.602494 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.602404    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eebda8d-38b2-41a9-a859-9596f7a0ca6f" containerName="revoke-keys"
Jul 18 01:24:29.602550 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.602504    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2319f26-9438-4950-8d2b-0fd587e76876" containerName="istio-proxy"
Jul 18 01:24:29.605752 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.605731    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.608560 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.608539    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-46dd73e1-openshift-default-dockercfg-wqzx7\""
Jul 18 01:24:29.619083 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.619059    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"]
Jul 18 01:24:29.769085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769053    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577a5dd3-8525-4e33-a403-03d47b653f91-istiod-ca-cert\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769097    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-credential-socket\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769134    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-certs\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-envoy\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769179    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gknfp\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-kube-api-access-gknfp\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-data\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769464 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769280    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577a5dd3-8525-4e33-a403-03d47b653f91-istio-podinfo\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769464 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-socket\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.769464 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.769407    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-istio-token\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870581 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870503    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-istio-token\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870581 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870565    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577a5dd3-8525-4e33-a403-03d47b653f91-istiod-ca-cert\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870610    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-credential-socket\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870668    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-certs\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870705    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-envoy\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gknfp\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-kube-api-access-gknfp\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870779    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-data\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.870840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870830    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577a5dd3-8525-4e33-a403-03d47b653f91-istio-podinfo\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.871174 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.870874    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-socket\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.871174 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.871022    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-credential-socket\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.871272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.871253    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-data\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.871315 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.871291    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-socket\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.871423 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.871402    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577a5dd3-8525-4e33-a403-03d47b653f91-istiod-ca-cert\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.871489 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.871409    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-certs\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.873568 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.873541    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-envoy\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.873896 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.873876    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577a5dd3-8525-4e33-a403-03d47b653f91-istio-podinfo\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.881692 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.881665    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-istio-token\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.882358 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.881998    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gknfp\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-kube-api-access-gknfp\") pod \"e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") " pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:29.915570 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:29.915542    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:30.052723 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.052696    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"]
Jul 18 01:24:30.054583 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:24:30.054555    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577a5dd3_8525_4e33_a403_03d47b653f91.slice/crio-7d289f819d88e047761fd1543c1a589e57fbf8b491f0077dd33a5e9ffd7f1f74 WatchSource:0}: Error finding container 7d289f819d88e047761fd1543c1a589e57fbf8b491f0077dd33a5e9ffd7f1f74: Status 404 returned error can't find the container with id 7d289f819d88e047761fd1543c1a589e57fbf8b491f0077dd33a5e9ffd7f1f74
Jul 18 01:24:30.057194 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.057154    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:24:30.057288 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.057237    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:24:30.057288 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.057278    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:24:30.759960 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.759924    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" event={"ID":"577a5dd3-8525-4e33-a403-03d47b653f91","Type":"ContainerStarted","Data":"9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093"}
Jul 18 01:24:30.760397 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.759973    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" event={"ID":"577a5dd3-8525-4e33-a403-03d47b653f91","Type":"ContainerStarted","Data":"7d289f819d88e047761fd1543c1a589e57fbf8b491f0077dd33a5e9ffd7f1f74"}
Jul 18 01:24:30.780414 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.780367    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" podStartSLOduration=1.780352425 podStartE2EDuration="1.780352425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:30.779117713 +0000 UTC m=+2626.905052693" watchObservedRunningTime="2026-07-18 01:24:30.780352425 +0000 UTC m=+2626.906287403"
Jul 18 01:24:30.916127 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:30.916096    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:31.921669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:31.921640    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:32.770501 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:32.770464    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:32.771363 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:32.771343    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:36.331198 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:36.331165    2554 scope.go:117] "RemoveContainer" containerID="5bcc770dcb511814d21c9f1cfe1f13816d56b9e6aaa3fde11c244e40dd11e293"
Jul 18 01:24:36.791510 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:36.791469    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" event={"ID":"52713d76-78c6-4eae-8aca-9dd5b4cff7e3","Type":"ContainerStarted","Data":"1c7f7e70d600d8825410a1232e867950b6991cf566f0219acc66cc7752cfdd67"}
Jul 18 01:24:36.806957 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:36.806906    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" podStartSLOduration=20.806890338 podStartE2EDuration="20.806890338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:36.806134869 +0000 UTC m=+2632.932069848" watchObservedRunningTime="2026-07-18 01:24:36.806890338 +0000 UTC m=+2632.932825339"
Jul 18 01:24:37.798872 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:37.798838    2554 generic.go:356] "Generic (PLEG): container finished" podID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerID="1c7f7e70d600d8825410a1232e867950b6991cf566f0219acc66cc7752cfdd67" exitCode=6
Jul 18 01:24:37.799327 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:37.798932    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" event={"ID":"52713d76-78c6-4eae-8aca-9dd5b4cff7e3","Type":"ContainerDied","Data":"1c7f7e70d600d8825410a1232e867950b6991cf566f0219acc66cc7752cfdd67"}
Jul 18 01:24:37.799327 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:37.799010    2554 scope.go:117] "RemoveContainer" containerID="5bcc770dcb511814d21c9f1cfe1f13816d56b9e6aaa3fde11c244e40dd11e293"
Jul 18 01:24:37.799444 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:37.799367    2554 scope.go:117] "RemoveContainer" containerID="1c7f7e70d600d8825410a1232e867950b6991cf566f0219acc66cc7752cfdd67"
Jul 18 01:24:37.799610 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:37.799591    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t_odh-ai-gateway-infra(52713d76-78c6-4eae-8aca-9dd5b4cff7e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3"
Jul 18 01:24:37.822799 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:37.822775    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"]
Jul 18 01:24:38.939006 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.938979    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"
Jul 18 01:24:38.959072 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.959045    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713d76-78c6-4eae-8aca-9dd5b4cff7e3-maas-api-service-ca\") pod \"52713d76-78c6-4eae-8aca-9dd5b4cff7e3\" (UID: \"52713d76-78c6-4eae-8aca-9dd5b4cff7e3\") "
Jul 18 01:24:38.959511 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.959477    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52713d76-78c6-4eae-8aca-9dd5b4cff7e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52713d76-78c6-4eae-8aca-9dd5b4cff7e3" (UID: "52713d76-78c6-4eae-8aca-9dd5b4cff7e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:24:38.971546 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.971520    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"]
Jul 18 01:24:38.971984 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.971970    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.972035 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.971988    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.972035 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.971998    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.972035 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.972004    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.972035 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.972013    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.972035 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.972018    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.972195 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.972112    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.972195 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.972125    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:24:38.975676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.975657    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"
Jul 18 01:24:38.982401 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:38.982377    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"]
Jul 18 01:24:39.063597 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.063492    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2298c440-03e5-42b6-8e05-2d4b8768f804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg\" (UID: \"2298c440-03e5-42b6-8e05-2d4b8768f804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"
Jul 18 01:24:39.064016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.063991    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713d76-78c6-4eae-8aca-9dd5b4cff7e3-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:24:39.164592 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.164557    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2298c440-03e5-42b6-8e05-2d4b8768f804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg\" (UID: \"2298c440-03e5-42b6-8e05-2d4b8768f804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"
Jul 18 01:24:39.165269 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.165246    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2298c440-03e5-42b6-8e05-2d4b8768f804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg\" (UID: \"2298c440-03e5-42b6-8e05-2d4b8768f804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"
Jul 18 01:24:39.288904 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.288860    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"
Jul 18 01:24:39.538537 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.535911    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"]
Jul 18 01:24:39.540771 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:24:39.540737    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2298c440_03e5_42b6_8e05_2d4b8768f804.slice/crio-c55bad0b21a1a99a05fbc164e964de1f26ae6508e310595b84692b1c338eb7cd WatchSource:0}: Error finding container c55bad0b21a1a99a05fbc164e964de1f26ae6508e310595b84692b1c338eb7cd: Status 404 returned error can't find the container with id c55bad0b21a1a99a05fbc164e964de1f26ae6508e310595b84692b1c338eb7cd
Jul 18 01:24:39.814635 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.814579    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t" event={"ID":"52713d76-78c6-4eae-8aca-9dd5b4cff7e3","Type":"ContainerDied","Data":"ef2bc0c98d36b67ba30ea225c894979c01660165ee6ee8071ac7d7ea439e8fdc"}
Jul 18 01:24:39.814915 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.814897    2554 scope.go:117] "RemoveContainer" containerID="1c7f7e70d600d8825410a1232e867950b6991cf566f0219acc66cc7752cfdd67"
Jul 18 01:24:39.815092 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.814807    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"
Jul 18 01:24:39.817606 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.817582    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" event={"ID":"2298c440-03e5-42b6-8e05-2d4b8768f804","Type":"ContainerStarted","Data":"c55bad0b21a1a99a05fbc164e964de1f26ae6508e310595b84692b1c338eb7cd"}
Jul 18 01:24:39.852889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.852864    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"]
Jul 18 01:24:39.856262 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:39.856238    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-hwg2t"]
Jul 18 01:24:40.337044 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:40.337009    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" path="/var/lib/kubelet/pods/52713d76-78c6-4eae-8aca-9dd5b4cff7e3/volumes"
Jul 18 01:24:40.830482 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:40.830434    2554 generic.go:356] "Generic (PLEG): container finished" podID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerID="12d740ce968d4f3ff9c6846ffb823a6a0f0ac0d33b703b9597135e3a98c003f6" exitCode=6
Jul 18 01:24:40.830709 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:40.830501    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" event={"ID":"2298c440-03e5-42b6-8e05-2d4b8768f804","Type":"ContainerDied","Data":"12d740ce968d4f3ff9c6846ffb823a6a0f0ac0d33b703b9597135e3a98c003f6"}
Jul 18 01:24:40.830967 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:40.830940    2554 scope.go:117] "RemoveContainer" containerID="12d740ce968d4f3ff9c6846ffb823a6a0f0ac0d33b703b9597135e3a98c003f6"
Jul 18 01:24:41.839069 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:41.839043    2554 generic.go:356] "Generic (PLEG): container finished" podID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerID="861aa1711d351be80a3061d112b53cadeb40046870f045bb2fd756ef9268464a" exitCode=6
Jul 18 01:24:41.839422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:41.839130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" event={"ID":"2298c440-03e5-42b6-8e05-2d4b8768f804","Type":"ContainerDied","Data":"861aa1711d351be80a3061d112b53cadeb40046870f045bb2fd756ef9268464a"}
Jul 18 01:24:41.839422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:41.839194    2554 scope.go:117] "RemoveContainer" containerID="12d740ce968d4f3ff9c6846ffb823a6a0f0ac0d33b703b9597135e3a98c003f6"
Jul 18 01:24:41.839422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:41.839316    2554 scope.go:117] "RemoveContainer" containerID="861aa1711d351be80a3061d112b53cadeb40046870f045bb2fd756ef9268464a"
Jul 18 01:24:41.839583 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:41.839559    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg_odh-ai-gateway-infra(2298c440-03e5-42b6-8e05-2d4b8768f804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804"
Jul 18 01:24:42.846637 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:42.846586    2554 scope.go:117] "RemoveContainer" containerID="861aa1711d351be80a3061d112b53cadeb40046870f045bb2fd756ef9268464a"
Jul 18 01:24:42.847037 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:42.846898    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg_odh-ai-gateway-infra(2298c440-03e5-42b6-8e05-2d4b8768f804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804"
Jul 18 01:24:46.871338 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:46.871307    2554 generic.go:356] "Generic (PLEG): container finished" podID="577a5dd3-8525-4e33-a403-03d47b653f91" containerID="9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093" exitCode=0
Jul 18 01:24:46.871797 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:46.871386    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" event={"ID":"577a5dd3-8525-4e33-a403-03d47b653f91","Type":"ContainerDied","Data":"9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093"}
Jul 18 01:24:46.871850 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:46.871832    2554 scope.go:117] "RemoveContainer" containerID="9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093"
Jul 18 01:24:46.872851 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:46.872823    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:24:46.872927 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:46.872909    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:24:46.872969 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:46.872960    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:24:47.770797 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:47.770761    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:47.878707 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:47.878670    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" event={"ID":"577a5dd3-8525-4e33-a403-03d47b653f91","Type":"ContainerStarted","Data":"e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214"}
Jul 18 01:24:47.916422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:47.916385    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:47.917530 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:47.917487    2554 patch_prober.go:28] interesting pod/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Jul 18 01:24:47.917677 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:47.917556    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Jul 18 01:24:48.916341 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:48.916301    2554 patch_prober.go:28] interesting pod/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Jul 18 01:24:48.916934 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:48.916904    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Jul 18 01:24:49.916037 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:49.915992    2554 patch_prober.go:28] interesting pod/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Jul 18 01:24:49.916252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:49.916060    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Jul 18 01:24:50.916918 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:50.916875    2554 patch_prober.go:28] interesting pod/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Jul 18 01:24:50.917441 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:50.916965    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Jul 18 01:24:51.920135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:51.920100    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:51.920547 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:51.920361    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:51.921185 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:51.921167    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:24:57.330701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:57.330668    2554 scope.go:117] "RemoveContainer" containerID="861aa1711d351be80a3061d112b53cadeb40046870f045bb2fd756ef9268464a"
Jul 18 01:24:57.932372 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:57.932288    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" event={"ID":"2298c440-03e5-42b6-8e05-2d4b8768f804","Type":"ContainerStarted","Data":"38a0fad9f6bac8646ec0b52e6748c41c298116fcc3be285dbf2c3b88cfd35eef"}
Jul 18 01:24:57.947846 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:57.947798    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" podStartSLOduration=19.947783965 podStartE2EDuration="19.947783965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:57.945854032 +0000 UTC m=+2654.071789011" watchObservedRunningTime="2026-07-18 01:24:57.947783965 +0000 UTC m=+2654.073719015"
Jul 18 01:24:58.939073 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:58.939038    2554 generic.go:356] "Generic (PLEG): container finished" podID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerID="38a0fad9f6bac8646ec0b52e6748c41c298116fcc3be285dbf2c3b88cfd35eef" exitCode=6
Jul 18 01:24:58.939493 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:58.939110    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" event={"ID":"2298c440-03e5-42b6-8e05-2d4b8768f804","Type":"ContainerDied","Data":"38a0fad9f6bac8646ec0b52e6748c41c298116fcc3be285dbf2c3b88cfd35eef"}
Jul 18 01:24:58.939493 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:58.939169    2554 scope.go:117] "RemoveContainer" containerID="861aa1711d351be80a3061d112b53cadeb40046870f045bb2fd756ef9268464a"
Jul 18 01:24:58.939493 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:58.939442    2554 scope.go:117] "RemoveContainer" containerID="38a0fad9f6bac8646ec0b52e6748c41c298116fcc3be285dbf2c3b88cfd35eef"
Jul 18 01:24:58.939774 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:24:58.939750    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg_odh-ai-gateway-infra(2298c440-03e5-42b6-8e05-2d4b8768f804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804"
Jul 18 01:24:58.955716 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:24:58.955690    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"]
Jul 18 01:25:00.082703 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.082680    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"
Jul 18 01:25:00.170631 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.170591    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2298c440-03e5-42b6-8e05-2d4b8768f804-maas-api-service-ca\") pod \"2298c440-03e5-42b6-8e05-2d4b8768f804\" (UID: \"2298c440-03e5-42b6-8e05-2d4b8768f804\") "
Jul 18 01:25:00.170948 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.170925    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2298c440-03e5-42b6-8e05-2d4b8768f804-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2298c440-03e5-42b6-8e05-2d4b8768f804" (UID: "2298c440-03e5-42b6-8e05-2d4b8768f804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:25:00.271526 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.271497    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2298c440-03e5-42b6-8e05-2d4b8768f804-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:25:00.951950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.951920    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"
Jul 18 01:25:00.952136 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.951916    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg" event={"ID":"2298c440-03e5-42b6-8e05-2d4b8768f804","Type":"ContainerDied","Data":"c55bad0b21a1a99a05fbc164e964de1f26ae6508e310595b84692b1c338eb7cd"}
Jul 18 01:25:00.952136 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.952055    2554 scope.go:117] "RemoveContainer" containerID="38a0fad9f6bac8646ec0b52e6748c41c298116fcc3be285dbf2c3b88cfd35eef"
Jul 18 01:25:00.972259 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.972232    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"]
Jul 18 01:25:00.976252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:00.976232    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2zwlg"]
Jul 18 01:25:02.131255 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131208    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"]
Jul 18 01:25:02.131672 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131605    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:02.131672 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131631    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:02.131672 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131661    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:02.131672 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131666    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:02.131817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131728    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:02.131817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131737    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:02.131817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.131743    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52713d76-78c6-4eae-8aca-9dd5b4cff7e3" containerName="revoke-keys"
Jul 18 01:25:02.136456 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.136438    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"
Jul 18 01:25:02.139168 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.139146    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:25:02.140349 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.140324    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"]
Jul 18 01:25:02.291546 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.291510    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb719bc-aa7b-41de-822a-4caea179de32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794\" (UID: \"bdb719bc-aa7b-41de-822a-4caea179de32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"
Jul 18 01:25:02.339309 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.339274    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" path="/var/lib/kubelet/pods/2298c440-03e5-42b6-8e05-2d4b8768f804/volumes"
Jul 18 01:25:02.393114 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.393024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb719bc-aa7b-41de-822a-4caea179de32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794\" (UID: \"bdb719bc-aa7b-41de-822a-4caea179de32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"
Jul 18 01:25:02.393652 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.393632    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb719bc-aa7b-41de-822a-4caea179de32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794\" (UID: \"bdb719bc-aa7b-41de-822a-4caea179de32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"
Jul 18 01:25:02.448948 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.448916    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"
Jul 18 01:25:02.581289 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.581262    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"]
Jul 18 01:25:02.585348 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:25:02.585252    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb719bc_aa7b_41de_822a_4caea179de32.slice/crio-aaf4025f0ee68f629e6b5b65ae6def28ce9212b7352abc3461f9ee606ad58306 WatchSource:0}: Error finding container aaf4025f0ee68f629e6b5b65ae6def28ce9212b7352abc3461f9ee606ad58306: Status 404 returned error can't find the container with id aaf4025f0ee68f629e6b5b65ae6def28ce9212b7352abc3461f9ee606ad58306
Jul 18 01:25:02.964781 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.964739    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" event={"ID":"bdb719bc-aa7b-41de-822a-4caea179de32","Type":"ContainerStarted","Data":"b5fd672e493a7a7941d657acab4ef71078b527e993b3442f579bd0b6b53eb74a"}
Jul 18 01:25:02.964927 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.964803    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" event={"ID":"bdb719bc-aa7b-41de-822a-4caea179de32","Type":"ContainerStarted","Data":"aaf4025f0ee68f629e6b5b65ae6def28ce9212b7352abc3461f9ee606ad58306"}
Jul 18 01:25:02.981989 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:02.981937    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" podStartSLOduration=0.981921566 podStartE2EDuration="981.921566ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:25:02.979233257 +0000 UTC m=+2659.105168237" watchObservedRunningTime="2026-07-18 01:25:02.981921566 +0000 UTC m=+2659.107856545"
Jul 18 01:25:03.970969 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:03.970934    2554 generic.go:356] "Generic (PLEG): container finished" podID="bdb719bc-aa7b-41de-822a-4caea179de32" containerID="b5fd672e493a7a7941d657acab4ef71078b527e993b3442f579bd0b6b53eb74a" exitCode=6
Jul 18 01:25:03.971365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:03.971002    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" event={"ID":"bdb719bc-aa7b-41de-822a-4caea179de32","Type":"ContainerDied","Data":"b5fd672e493a7a7941d657acab4ef71078b527e993b3442f579bd0b6b53eb74a"}
Jul 18 01:25:03.971365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:03.971237    2554 scope.go:117] "RemoveContainer" containerID="b5fd672e493a7a7941d657acab4ef71078b527e993b3442f579bd0b6b53eb74a"
Jul 18 01:25:04.978124 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:04.978026    2554 generic.go:356] "Generic (PLEG): container finished" podID="bdb719bc-aa7b-41de-822a-4caea179de32" containerID="510229b8ab92ce838bf3f303a1ad7d7784008c3f6214d2a1e21015d34d53b0d3" exitCode=6
Jul 18 01:25:04.978124 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:04.978109    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" event={"ID":"bdb719bc-aa7b-41de-822a-4caea179de32","Type":"ContainerDied","Data":"510229b8ab92ce838bf3f303a1ad7d7784008c3f6214d2a1e21015d34d53b0d3"}
Jul 18 01:25:04.978667 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:04.978173    2554 scope.go:117] "RemoveContainer" containerID="b5fd672e493a7a7941d657acab4ef71078b527e993b3442f579bd0b6b53eb74a"
Jul 18 01:25:04.978667 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:04.978429    2554 scope.go:117] "RemoveContainer" containerID="510229b8ab92ce838bf3f303a1ad7d7784008c3f6214d2a1e21015d34d53b0d3"
Jul 18 01:25:04.978667 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:04.978638    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794_odh-ai-gateway-infra(bdb719bc-aa7b-41de-822a-4caea179de32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" podUID="bdb719bc-aa7b-41de-822a-4caea179de32"
Jul 18 01:25:05.991689 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:05.991656    2554 scope.go:117] "RemoveContainer" containerID="510229b8ab92ce838bf3f303a1ad7d7784008c3f6214d2a1e21015d34d53b0d3"
Jul 18 01:25:05.992134 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:05.991930    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794_odh-ai-gateway-infra(bdb719bc-aa7b-41de-822a-4caea179de32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" podUID="bdb719bc-aa7b-41de-822a-4caea179de32"
Jul 18 01:25:10.485556 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.485517    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"]
Jul 18 01:25:10.486003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.485931    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:10.486003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.485943    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:10.486003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.486003    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2298c440-03e5-42b6-8e05-2d4b8768f804" containerName="revoke-keys"
Jul 18 01:25:10.490277 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.490259    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"
Jul 18 01:25:10.496109 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.496085    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"]
Jul 18 01:25:10.678154 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.678118    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5372c023-1251-414a-8872-693da4c98af0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz\" (UID: \"5372c023-1251-414a-8872-693da4c98af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"
Jul 18 01:25:10.778885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.778847    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5372c023-1251-414a-8872-693da4c98af0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz\" (UID: \"5372c023-1251-414a-8872-693da4c98af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"
Jul 18 01:25:10.779458 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.779435    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5372c023-1251-414a-8872-693da4c98af0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz\" (UID: \"5372c023-1251-414a-8872-693da4c98af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"
Jul 18 01:25:10.802576 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.802550    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"
Jul 18 01:25:10.935748 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:10.935723    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"]
Jul 18 01:25:10.937683 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:25:10.937659    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5372c023_1251_414a_8872_693da4c98af0.slice/crio-69475ecec75c136c65d14932f0a3a7408b452cc61e9baef2a2bd6ae6d5308e00 WatchSource:0}: Error finding container 69475ecec75c136c65d14932f0a3a7408b452cc61e9baef2a2bd6ae6d5308e00: Status 404 returned error can't find the container with id 69475ecec75c136c65d14932f0a3a7408b452cc61e9baef2a2bd6ae6d5308e00
Jul 18 01:25:11.017017 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:11.016986    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz" event={"ID":"5372c023-1251-414a-8872-693da4c98af0","Type":"ContainerStarted","Data":"69475ecec75c136c65d14932f0a3a7408b452cc61e9baef2a2bd6ae6d5308e00"}
Jul 18 01:25:12.023550 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:12.023515    2554 generic.go:356] "Generic (PLEG): container finished" podID="5372c023-1251-414a-8872-693da4c98af0" containerID="c91b109268c42c00b308622b110384ce80e379c89b0ac6c71dca433fcf34326d" exitCode=0
Jul 18 01:25:12.023987 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:12.023596    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz" event={"ID":"5372c023-1251-414a-8872-693da4c98af0","Type":"ContainerDied","Data":"c91b109268c42c00b308622b110384ce80e379c89b0ac6c71dca433fcf34326d"}
Jul 18 01:25:13.168126 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:13.168097    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"
Jul 18 01:25:13.306059 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:13.305977    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5372c023-1251-414a-8872-693da4c98af0-maas-api-service-ca\") pod \"5372c023-1251-414a-8872-693da4c98af0\" (UID: \"5372c023-1251-414a-8872-693da4c98af0\") "
Jul 18 01:25:13.306371 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:13.306350    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5372c023-1251-414a-8872-693da4c98af0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5372c023-1251-414a-8872-693da4c98af0" (UID: "5372c023-1251-414a-8872-693da4c98af0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:25:13.406830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:13.406803    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5372c023-1251-414a-8872-693da4c98af0-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:25:14.035380 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:14.035340    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz" event={"ID":"5372c023-1251-414a-8872-693da4c98af0","Type":"ContainerDied","Data":"69475ecec75c136c65d14932f0a3a7408b452cc61e9baef2a2bd6ae6d5308e00"}
Jul 18 01:25:14.035380 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:14.035364    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz"
Jul 18 01:25:14.035696 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:14.035418    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69475ecec75c136c65d14932f0a3a7408b452cc61e9baef2a2bd6ae6d5308e00"
Jul 18 01:25:15.040542 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.040510    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"]
Jul 18 01:25:15.040964 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.040923    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5372c023-1251-414a-8872-693da4c98af0" containerName="revoke-keys"
Jul 18 01:25:15.040964 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.040935    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5372c023-1251-414a-8872-693da4c98af0" containerName="revoke-keys"
Jul 18 01:25:15.041043 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.040998    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5372c023-1251-414a-8872-693da4c98af0" containerName="revoke-keys"
Jul 18 01:25:15.043964 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.043947    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"
Jul 18 01:25:15.050401 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.050376    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"]
Jul 18 01:25:15.119647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.119589    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81bef3f-d169-43e4-bcb5-8de8e859cccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-2ffzt\" (UID: \"f81bef3f-d169-43e4-bcb5-8de8e859cccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"
Jul 18 01:25:15.220348 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.220310    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81bef3f-d169-43e4-bcb5-8de8e859cccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-2ffzt\" (UID: \"f81bef3f-d169-43e4-bcb5-8de8e859cccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"
Jul 18 01:25:15.220997 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.220971    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81bef3f-d169-43e4-bcb5-8de8e859cccf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-2ffzt\" (UID: \"f81bef3f-d169-43e4-bcb5-8de8e859cccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"
Jul 18 01:25:15.356891 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.356797    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"
Jul 18 01:25:15.493039 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:15.493005    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"]
Jul 18 01:25:15.495816 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:25:15.495787    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81bef3f_d169_43e4_bcb5_8de8e859cccf.slice/crio-590efa72e01b67c2678f5f21e71b3bd25b83f35957b3951e76f5662df08bd4e1 WatchSource:0}: Error finding container 590efa72e01b67c2678f5f21e71b3bd25b83f35957b3951e76f5662df08bd4e1: Status 404 returned error can't find the container with id 590efa72e01b67c2678f5f21e71b3bd25b83f35957b3951e76f5662df08bd4e1
Jul 18 01:25:16.047207 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:16.047165    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" event={"ID":"f81bef3f-d169-43e4-bcb5-8de8e859cccf","Type":"ContainerStarted","Data":"c28afbe26804047be89705aca436fe9d8eaf85a99cfae6cfee030b502528a96e"}
Jul 18 01:25:16.047714 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:16.047233    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" event={"ID":"f81bef3f-d169-43e4-bcb5-8de8e859cccf","Type":"ContainerStarted","Data":"590efa72e01b67c2678f5f21e71b3bd25b83f35957b3951e76f5662df08bd4e1"}
Jul 18 01:25:16.063914 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:16.063867    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" podStartSLOduration=1.063851961 podStartE2EDuration="1.063851961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:25:16.060834858 +0000 UTC m=+2672.186769849" watchObservedRunningTime="2026-07-18 01:25:16.063851961 +0000 UTC m=+2672.189786939"
Jul 18 01:25:17.046061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.046028    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"]
Jul 18 01:25:17.049643 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.049609    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"
Jul 18 01:25:17.054224 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.054195    2554 generic.go:356] "Generic (PLEG): container finished" podID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerID="c28afbe26804047be89705aca436fe9d8eaf85a99cfae6cfee030b502528a96e" exitCode=6
Jul 18 01:25:17.054347 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.054248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" event={"ID":"f81bef3f-d169-43e4-bcb5-8de8e859cccf","Type":"ContainerDied","Data":"c28afbe26804047be89705aca436fe9d8eaf85a99cfae6cfee030b502528a96e"}
Jul 18 01:25:17.054491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.054473    2554 scope.go:117] "RemoveContainer" containerID="c28afbe26804047be89705aca436fe9d8eaf85a99cfae6cfee030b502528a96e"
Jul 18 01:25:17.058373 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.058346    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"]
Jul 18 01:25:17.139884 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.139840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157a57c0-c6ff-493f-a8dd-523ebf10ec13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk\" (UID: \"157a57c0-c6ff-493f-a8dd-523ebf10ec13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"
Jul 18 01:25:17.240412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.240372    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157a57c0-c6ff-493f-a8dd-523ebf10ec13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk\" (UID: \"157a57c0-c6ff-493f-a8dd-523ebf10ec13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"
Jul 18 01:25:17.241007 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.240986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157a57c0-c6ff-493f-a8dd-523ebf10ec13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk\" (UID: \"157a57c0-c6ff-493f-a8dd-523ebf10ec13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"
Jul 18 01:25:17.331152 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.331120    2554 scope.go:117] "RemoveContainer" containerID="510229b8ab92ce838bf3f303a1ad7d7784008c3f6214d2a1e21015d34d53b0d3"
Jul 18 01:25:17.362384 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.362359    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"
Jul 18 01:25:17.501241 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:17.501215    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"]
Jul 18 01:25:17.503602 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:25:17.503568    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157a57c0_c6ff_493f_a8dd_523ebf10ec13.slice/crio-2017ae2bfd9edea966f6a329c9313bef20c542884aaa7d9caf6e4fe415941c7a WatchSource:0}: Error finding container 2017ae2bfd9edea966f6a329c9313bef20c542884aaa7d9caf6e4fe415941c7a: Status 404 returned error can't find the container with id 2017ae2bfd9edea966f6a329c9313bef20c542884aaa7d9caf6e4fe415941c7a
Jul 18 01:25:18.060961 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.060931    2554 generic.go:356] "Generic (PLEG): container finished" podID="bdb719bc-aa7b-41de-822a-4caea179de32" containerID="44cee30a6b64e42c225a861672455f0cab5d859a2c8832aeb87bf348efd3a915" exitCode=6
Jul 18 01:25:18.061431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.061010    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" event={"ID":"bdb719bc-aa7b-41de-822a-4caea179de32","Type":"ContainerDied","Data":"44cee30a6b64e42c225a861672455f0cab5d859a2c8832aeb87bf348efd3a915"}
Jul 18 01:25:18.061431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.061081    2554 scope.go:117] "RemoveContainer" containerID="510229b8ab92ce838bf3f303a1ad7d7784008c3f6214d2a1e21015d34d53b0d3"
Jul 18 01:25:18.061431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.061359    2554 scope.go:117] "RemoveContainer" containerID="44cee30a6b64e42c225a861672455f0cab5d859a2c8832aeb87bf348efd3a915"
Jul 18 01:25:18.061594 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:18.061574    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794_odh-ai-gateway-infra(bdb719bc-aa7b-41de-822a-4caea179de32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" podUID="bdb719bc-aa7b-41de-822a-4caea179de32"
Jul 18 01:25:18.063263 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.063239    2554 generic.go:356] "Generic (PLEG): container finished" podID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerID="4749981ddfce91daf9f513e83089ce992a57b74f0b8d8cf784afbb870e3e75e1" exitCode=6
Jul 18 01:25:18.063348 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.063317    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" event={"ID":"f81bef3f-d169-43e4-bcb5-8de8e859cccf","Type":"ContainerDied","Data":"4749981ddfce91daf9f513e83089ce992a57b74f0b8d8cf784afbb870e3e75e1"}
Jul 18 01:25:18.063526 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.063510    2554 scope.go:117] "RemoveContainer" containerID="4749981ddfce91daf9f513e83089ce992a57b74f0b8d8cf784afbb870e3e75e1"
Jul 18 01:25:18.063737 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:18.063719    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-2ffzt_odh-ai-gateway-infra(f81bef3f-d169-43e4-bcb5-8de8e859cccf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf"
Jul 18 01:25:18.064807 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.064778    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" event={"ID":"157a57c0-c6ff-493f-a8dd-523ebf10ec13","Type":"ContainerStarted","Data":"4c5cfd9fefb5ea4045ccbfd9c772a6769b226527374cae0e6e05ed6c8c67408a"}
Jul 18 01:25:18.064896 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.064829    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" event={"ID":"157a57c0-c6ff-493f-a8dd-523ebf10ec13","Type":"ContainerStarted","Data":"2017ae2bfd9edea966f6a329c9313bef20c542884aaa7d9caf6e4fe415941c7a"}
Jul 18 01:25:18.081290 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.081270    2554 scope.go:117] "RemoveContainer" containerID="c28afbe26804047be89705aca436fe9d8eaf85a99cfae6cfee030b502528a96e"
Jul 18 01:25:18.091250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:18.091208    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" podStartSLOduration=1.091191858 podStartE2EDuration="1.091191858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:25:18.089121791 +0000 UTC m=+2674.215056767" watchObservedRunningTime="2026-07-18 01:25:18.091191858 +0000 UTC m=+2674.217126837"
Jul 18 01:25:19.072074 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.072042    2554 scope.go:117] "RemoveContainer" containerID="4749981ddfce91daf9f513e83089ce992a57b74f0b8d8cf784afbb870e3e75e1"
Jul 18 01:25:19.072522 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:19.072288    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-2ffzt_odh-ai-gateway-infra(f81bef3f-d169-43e4-bcb5-8de8e859cccf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf"
Jul 18 01:25:19.073254 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.073225    2554 generic.go:356] "Generic (PLEG): container finished" podID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerID="4c5cfd9fefb5ea4045ccbfd9c772a6769b226527374cae0e6e05ed6c8c67408a" exitCode=6
Jul 18 01:25:19.073372 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.073302    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" event={"ID":"157a57c0-c6ff-493f-a8dd-523ebf10ec13","Type":"ContainerDied","Data":"4c5cfd9fefb5ea4045ccbfd9c772a6769b226527374cae0e6e05ed6c8c67408a"}
Jul 18 01:25:19.073490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.073475    2554 scope.go:117] "RemoveContainer" containerID="4c5cfd9fefb5ea4045ccbfd9c772a6769b226527374cae0e6e05ed6c8c67408a"
Jul 18 01:25:19.086098 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.086072    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"]
Jul 18 01:25:19.227227 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.227202    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"
Jul 18 01:25:19.260811 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.260782    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb719bc-aa7b-41de-822a-4caea179de32-maas-api-service-ca\") pod \"bdb719bc-aa7b-41de-822a-4caea179de32\" (UID: \"bdb719bc-aa7b-41de-822a-4caea179de32\") "
Jul 18 01:25:19.261135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.261112    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb719bc-aa7b-41de-822a-4caea179de32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdb719bc-aa7b-41de-822a-4caea179de32" (UID: "bdb719bc-aa7b-41de-822a-4caea179de32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:25:19.363193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:19.363023    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb719bc-aa7b-41de-822a-4caea179de32-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:25:20.082991 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.082960    2554 generic.go:356] "Generic (PLEG): container finished" podID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerID="0bd6ea9e885807e4f16957ab9c07ad4778f6a2b5429ec5223255830ad91d8cc2" exitCode=6
Jul 18 01:25:20.083443 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.083038    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" event={"ID":"157a57c0-c6ff-493f-a8dd-523ebf10ec13","Type":"ContainerDied","Data":"0bd6ea9e885807e4f16957ab9c07ad4778f6a2b5429ec5223255830ad91d8cc2"}
Jul 18 01:25:20.083443 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.083104    2554 scope.go:117] "RemoveContainer" containerID="4c5cfd9fefb5ea4045ccbfd9c772a6769b226527374cae0e6e05ed6c8c67408a"
Jul 18 01:25:20.083443 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.083255    2554 scope.go:117] "RemoveContainer" containerID="0bd6ea9e885807e4f16957ab9c07ad4778f6a2b5429ec5223255830ad91d8cc2"
Jul 18 01:25:20.083653 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:20.083549    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk_odh-ai-gateway-infra(157a57c0-c6ff-493f-a8dd-523ebf10ec13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13"
Jul 18 01:25:20.084548 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.084526    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794" event={"ID":"bdb719bc-aa7b-41de-822a-4caea179de32","Type":"ContainerDied","Data":"aaf4025f0ee68f629e6b5b65ae6def28ce9212b7352abc3461f9ee606ad58306"}
Jul 18 01:25:20.084653 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.084535    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"
Jul 18 01:25:20.095557 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.095535    2554 scope.go:117] "RemoveContainer" containerID="44cee30a6b64e42c225a861672455f0cab5d859a2c8832aeb87bf348efd3a915"
Jul 18 01:25:20.110258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.110232    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"]
Jul 18 01:25:20.114397 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.114369    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-8p794"]
Jul 18 01:25:20.336979 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:20.336903    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" path="/var/lib/kubelet/pods/bdb719bc-aa7b-41de-822a-4caea179de32/volumes"
Jul 18 01:25:21.092294 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:21.092266    2554 scope.go:117] "RemoveContainer" containerID="0bd6ea9e885807e4f16957ab9c07ad4778f6a2b5429ec5223255830ad91d8cc2"
Jul 18 01:25:21.092702 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:21.092484    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk_odh-ai-gateway-infra(157a57c0-c6ff-493f-a8dd-523ebf10ec13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13"
Jul 18 01:25:33.330599 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:33.330567    2554 scope.go:117] "RemoveContainer" containerID="4749981ddfce91daf9f513e83089ce992a57b74f0b8d8cf784afbb870e3e75e1"
Jul 18 01:25:34.160387 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:34.160355    2554 generic.go:356] "Generic (PLEG): container finished" podID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerID="94fbee7883f4d5fcf5f8f62ad2037584bc8020fb6a641e0c5a2328f906876b83" exitCode=6
Jul 18 01:25:34.160570 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:34.160414    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" event={"ID":"f81bef3f-d169-43e4-bcb5-8de8e859cccf","Type":"ContainerDied","Data":"94fbee7883f4d5fcf5f8f62ad2037584bc8020fb6a641e0c5a2328f906876b83"}
Jul 18 01:25:34.160570 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:34.160466    2554 scope.go:117] "RemoveContainer" containerID="4749981ddfce91daf9f513e83089ce992a57b74f0b8d8cf784afbb870e3e75e1"
Jul 18 01:25:34.160802 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:34.160784    2554 scope.go:117] "RemoveContainer" containerID="94fbee7883f4d5fcf5f8f62ad2037584bc8020fb6a641e0c5a2328f906876b83"
Jul 18 01:25:34.161089 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:34.161067    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-2ffzt_odh-ai-gateway-infra(f81bef3f-d169-43e4-bcb5-8de8e859cccf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf"
Jul 18 01:25:35.184768 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:35.184739    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"]
Jul 18 01:25:35.324659 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:35.324610    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"
Jul 18 01:25:35.330421 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:35.330403    2554 scope.go:117] "RemoveContainer" containerID="0bd6ea9e885807e4f16957ab9c07ad4778f6a2b5429ec5223255830ad91d8cc2"
Jul 18 01:25:35.422016 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:35.421983    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81bef3f-d169-43e4-bcb5-8de8e859cccf-maas-api-service-ca\") pod \"f81bef3f-d169-43e4-bcb5-8de8e859cccf\" (UID: \"f81bef3f-d169-43e4-bcb5-8de8e859cccf\") "
Jul 18 01:25:35.422362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:35.422337    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81bef3f-d169-43e4-bcb5-8de8e859cccf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f81bef3f-d169-43e4-bcb5-8de8e859cccf" (UID: "f81bef3f-d169-43e4-bcb5-8de8e859cccf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:25:35.523757 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:35.523726    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81bef3f-d169-43e4-bcb5-8de8e859cccf-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:25:36.173818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.173722    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt" event={"ID":"f81bef3f-d169-43e4-bcb5-8de8e859cccf","Type":"ContainerDied","Data":"590efa72e01b67c2678f5f21e71b3bd25b83f35957b3951e76f5662df08bd4e1"}
Jul 18 01:25:36.173818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.173748    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"
Jul 18 01:25:36.173818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.173794    2554 scope.go:117] "RemoveContainer" containerID="94fbee7883f4d5fcf5f8f62ad2037584bc8020fb6a641e0c5a2328f906876b83"
Jul 18 01:25:36.175970 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.175942    2554 generic.go:356] "Generic (PLEG): container finished" podID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerID="934c08d8a142b0908c6a999cf369c503f0280115cc69e6bed8be6cfa4651a81d" exitCode=6
Jul 18 01:25:36.176116 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.175977    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" event={"ID":"157a57c0-c6ff-493f-a8dd-523ebf10ec13","Type":"ContainerDied","Data":"934c08d8a142b0908c6a999cf369c503f0280115cc69e6bed8be6cfa4651a81d"}
Jul 18 01:25:36.176328 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.176313    2554 scope.go:117] "RemoveContainer" containerID="934c08d8a142b0908c6a999cf369c503f0280115cc69e6bed8be6cfa4651a81d"
Jul 18 01:25:36.176515 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:36.176496    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk_odh-ai-gateway-infra(157a57c0-c6ff-493f-a8dd-523ebf10ec13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13"
Jul 18 01:25:36.189310 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.189119    2554 scope.go:117] "RemoveContainer" containerID="0bd6ea9e885807e4f16957ab9c07ad4778f6a2b5429ec5223255830ad91d8cc2"
Jul 18 01:25:36.210250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.210224    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"]
Jul 18 01:25:36.212164 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.212142    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-2ffzt"]
Jul 18 01:25:36.336437 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:36.336403    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" path="/var/lib/kubelet/pods/f81bef3f-d169-43e4-bcb5-8de8e859cccf/volumes"
Jul 18 01:25:37.206137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:37.206087    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"]
Jul 18 01:25:37.344151 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:37.344120    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"
Jul 18 01:25:37.443268 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:37.443227    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157a57c0-c6ff-493f-a8dd-523ebf10ec13-maas-api-service-ca\") pod \"157a57c0-c6ff-493f-a8dd-523ebf10ec13\" (UID: \"157a57c0-c6ff-493f-a8dd-523ebf10ec13\") "
Jul 18 01:25:37.443677 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:37.443641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157a57c0-c6ff-493f-a8dd-523ebf10ec13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "157a57c0-c6ff-493f-a8dd-523ebf10ec13" (UID: "157a57c0-c6ff-493f-a8dd-523ebf10ec13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:25:37.544686 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:37.544646    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/157a57c0-c6ff-493f-a8dd-523ebf10ec13-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:25:38.190608 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:38.190577    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"
Jul 18 01:25:38.190840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:38.190609    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk" event={"ID":"157a57c0-c6ff-493f-a8dd-523ebf10ec13","Type":"ContainerDied","Data":"2017ae2bfd9edea966f6a329c9313bef20c542884aaa7d9caf6e4fe415941c7a"}
Jul 18 01:25:38.190840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:38.190691    2554 scope.go:117] "RemoveContainer" containerID="934c08d8a142b0908c6a999cf369c503f0280115cc69e6bed8be6cfa4651a81d"
Jul 18 01:25:38.212972 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:38.212936    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"]
Jul 18 01:25:38.218904 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:38.218865    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-x42sk"]
Jul 18 01:25:38.336372 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:38.336339    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" path="/var/lib/kubelet/pods/157a57c0-c6ff-493f-a8dd-523ebf10ec13/volumes"
Jul 18 01:25:39.018430 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018394    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"]
Jul 18 01:25:39.018814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018801    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018816    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018827    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018833    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018842    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018848    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018855    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018861    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.018867 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018869    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018874    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018880    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018886    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018895    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018901    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018910    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018916    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018979    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018987    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018992    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.018999    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.019005    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.019012    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.019113 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.019018    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="157a57c0-c6ff-493f-a8dd-523ebf10ec13" containerName="revoke-keys"
Jul 18 01:25:39.023388 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.023372    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"
Jul 18 01:25:39.026459 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.026436    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:25:39.028317 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.028297    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"]
Jul 18 01:25:39.161119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.161084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd53f45c-dffa-4ec1-be2e-c91a4a638427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7\" (UID: \"dd53f45c-dffa-4ec1-be2e-c91a4a638427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"
Jul 18 01:25:39.262119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.262086    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd53f45c-dffa-4ec1-be2e-c91a4a638427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7\" (UID: \"dd53f45c-dffa-4ec1-be2e-c91a4a638427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"
Jul 18 01:25:39.262859 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.262831    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd53f45c-dffa-4ec1-be2e-c91a4a638427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7\" (UID: \"dd53f45c-dffa-4ec1-be2e-c91a4a638427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"
Jul 18 01:25:39.336172 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.336078    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"
Jul 18 01:25:39.466437 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:39.466408    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"]
Jul 18 01:25:39.468562 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:25:39.468535    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd53f45c_dffa_4ec1_be2e_c91a4a638427.slice/crio-246a8a91bfb826c150ec4fc1ecf35469f04288d10f2a8300d8ae4a16b0bdce1c WatchSource:0}: Error finding container 246a8a91bfb826c150ec4fc1ecf35469f04288d10f2a8300d8ae4a16b0bdce1c: Status 404 returned error can't find the container with id 246a8a91bfb826c150ec4fc1ecf35469f04288d10f2a8300d8ae4a16b0bdce1c
Jul 18 01:25:40.203143 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:40.203107    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerID="176fd3dbd016a094662eee4290a00b20dc112f32e588d8f406ee208a5810076e" exitCode=6
Jul 18 01:25:40.203327 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:40.203194    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" event={"ID":"dd53f45c-dffa-4ec1-be2e-c91a4a638427","Type":"ContainerDied","Data":"176fd3dbd016a094662eee4290a00b20dc112f32e588d8f406ee208a5810076e"}
Jul 18 01:25:40.203327 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:40.203256    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" event={"ID":"dd53f45c-dffa-4ec1-be2e-c91a4a638427","Type":"ContainerStarted","Data":"246a8a91bfb826c150ec4fc1ecf35469f04288d10f2a8300d8ae4a16b0bdce1c"}
Jul 18 01:25:40.203423 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:40.203379    2554 scope.go:117] "RemoveContainer" containerID="176fd3dbd016a094662eee4290a00b20dc112f32e588d8f406ee208a5810076e"
Jul 18 01:25:41.211314 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:41.211218    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerID="39dd1c91329f79c91791153f54836152f7861e52cba540f93510baf3eae54078" exitCode=6
Jul 18 01:25:41.211730 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:41.211303    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" event={"ID":"dd53f45c-dffa-4ec1-be2e-c91a4a638427","Type":"ContainerDied","Data":"39dd1c91329f79c91791153f54836152f7861e52cba540f93510baf3eae54078"}
Jul 18 01:25:41.211730 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:41.211389    2554 scope.go:117] "RemoveContainer" containerID="176fd3dbd016a094662eee4290a00b20dc112f32e588d8f406ee208a5810076e"
Jul 18 01:25:41.211730 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:41.211478    2554 scope.go:117] "RemoveContainer" containerID="39dd1c91329f79c91791153f54836152f7861e52cba540f93510baf3eae54078"
Jul 18 01:25:41.211853 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:41.211783    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7_odh-ai-gateway-infra(dd53f45c-dffa-4ec1-be2e-c91a4a638427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427"
Jul 18 01:25:42.220104 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:42.220079    2554 scope.go:117] "RemoveContainer" containerID="39dd1c91329f79c91791153f54836152f7861e52cba540f93510baf3eae54078"
Jul 18 01:25:42.220489 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:42.220282    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7_odh-ai-gateway-infra(dd53f45c-dffa-4ec1-be2e-c91a4a638427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427"
Jul 18 01:25:54.334172 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:54.334135    2554 scope.go:117] "RemoveContainer" containerID="39dd1c91329f79c91791153f54836152f7861e52cba540f93510baf3eae54078"
Jul 18 01:25:55.284045 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:55.283956    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerID="1d5ddfca4f9164a9132d4c214f9abd9e880da58c769475885d269e649a1d94fd" exitCode=6
Jul 18 01:25:55.284045 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:55.284025    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" event={"ID":"dd53f45c-dffa-4ec1-be2e-c91a4a638427","Type":"ContainerDied","Data":"1d5ddfca4f9164a9132d4c214f9abd9e880da58c769475885d269e649a1d94fd"}
Jul 18 01:25:55.284272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:55.284088    2554 scope.go:117] "RemoveContainer" containerID="39dd1c91329f79c91791153f54836152f7861e52cba540f93510baf3eae54078"
Jul 18 01:25:55.284386 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:55.284370    2554 scope.go:117] "RemoveContainer" containerID="1d5ddfca4f9164a9132d4c214f9abd9e880da58c769475885d269e649a1d94fd"
Jul 18 01:25:55.284704 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:25:55.284683    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7_odh-ai-gateway-infra(dd53f45c-dffa-4ec1-be2e-c91a4a638427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427"
Jul 18 01:25:56.307849 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:56.307806    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"]
Jul 18 01:25:56.469942 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:56.469915    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"
Jul 18 01:25:56.525315 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:56.525281    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd53f45c-dffa-4ec1-be2e-c91a4a638427-maas-api-service-ca\") pod \"dd53f45c-dffa-4ec1-be2e-c91a4a638427\" (UID: \"dd53f45c-dffa-4ec1-be2e-c91a4a638427\") "
Jul 18 01:25:56.525685 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:56.525660    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd53f45c-dffa-4ec1-be2e-c91a4a638427-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd53f45c-dffa-4ec1-be2e-c91a4a638427" (UID: "dd53f45c-dffa-4ec1-be2e-c91a4a638427"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:25:56.626324 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:56.626226    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd53f45c-dffa-4ec1-be2e-c91a4a638427-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:25:57.297555 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:57.297523    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"
Jul 18 01:25:57.297555 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:57.297536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7" event={"ID":"dd53f45c-dffa-4ec1-be2e-c91a4a638427","Type":"ContainerDied","Data":"246a8a91bfb826c150ec4fc1ecf35469f04288d10f2a8300d8ae4a16b0bdce1c"}
Jul 18 01:25:57.297843 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:57.297646    2554 scope.go:117] "RemoveContainer" containerID="1d5ddfca4f9164a9132d4c214f9abd9e880da58c769475885d269e649a1d94fd"
Jul 18 01:25:57.319815 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:57.319784    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"]
Jul 18 01:25:57.325065 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:57.325041    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-zmgm7"]
Jul 18 01:25:58.336572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:25:58.336537    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" path="/var/lib/kubelet/pods/dd53f45c-dffa-4ec1-be2e-c91a4a638427/volumes"
Jul 18 01:26:17.107364 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107330    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"]
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107744    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107757    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107770    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107776    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107783    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107788    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107796    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.107796 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107802    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.108053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107862    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81bef3f-d169-43e4-bcb5-8de8e859cccf" containerName="revoke-keys"
Jul 18 01:26:17.108053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107870    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.108053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107877    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb719bc-aa7b-41de-822a-4caea179de32" containerName="revoke-keys"
Jul 18 01:26:17.108053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.107886    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:17.115669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.115640    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"
Jul 18 01:26:17.118655 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.118479    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:26:17.119451 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.119422    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"]
Jul 18 01:26:17.226280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.226239    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596ed215-8237-474a-afe7-703b817308b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk\" (UID: \"596ed215-8237-474a-afe7-703b817308b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"
Jul 18 01:26:17.327602 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.327562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596ed215-8237-474a-afe7-703b817308b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk\" (UID: \"596ed215-8237-474a-afe7-703b817308b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"
Jul 18 01:26:17.328383 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.328348    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596ed215-8237-474a-afe7-703b817308b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk\" (UID: \"596ed215-8237-474a-afe7-703b817308b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"
Jul 18 01:26:17.428476 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.428389    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"
Jul 18 01:26:17.570340 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.570284    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"]
Jul 18 01:26:17.572885 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:26:17.572848    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596ed215_8237_474a_afe7_703b817308b0.slice/crio-334968ad6fa90655489adb51ebe340b945a5dca44d98e126a360e397408ea5f3 WatchSource:0}: Error finding container 334968ad6fa90655489adb51ebe340b945a5dca44d98e126a360e397408ea5f3: Status 404 returned error can't find the container with id 334968ad6fa90655489adb51ebe340b945a5dca44d98e126a360e397408ea5f3
Jul 18 01:26:17.714057 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.713989    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"]
Jul 18 01:26:17.714216 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:17.714193    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy" containerID="cri-o://e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214" gracePeriod=30
Jul 18 01:26:18.402864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:18.402834    2554 generic.go:356] "Generic (PLEG): container finished" podID="596ed215-8237-474a-afe7-703b817308b0" containerID="e0cc5cd1c556d0ba3837d92289c1b2bf0d7e879d6ff755cfabe00ff0216d5a3a" exitCode=6
Jul 18 01:26:18.403250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:18.402920    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" event={"ID":"596ed215-8237-474a-afe7-703b817308b0","Type":"ContainerDied","Data":"e0cc5cd1c556d0ba3837d92289c1b2bf0d7e879d6ff755cfabe00ff0216d5a3a"}
Jul 18 01:26:18.403250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:18.402971    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" event={"ID":"596ed215-8237-474a-afe7-703b817308b0","Type":"ContainerStarted","Data":"334968ad6fa90655489adb51ebe340b945a5dca44d98e126a360e397408ea5f3"}
Jul 18 01:26:18.403250 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:18.403047    2554 scope.go:117] "RemoveContainer" containerID="e0cc5cd1c556d0ba3837d92289c1b2bf0d7e879d6ff755cfabe00ff0216d5a3a"
Jul 18 01:26:19.411046 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:19.410957    2554 generic.go:356] "Generic (PLEG): container finished" podID="596ed215-8237-474a-afe7-703b817308b0" containerID="6cbf144d807c0594cd45c7126365e32e9e93e50f9db5e16f942dd2eaa3c9ff70" exitCode=6
Jul 18 01:26:19.411046 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:19.410993    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" event={"ID":"596ed215-8237-474a-afe7-703b817308b0","Type":"ContainerDied","Data":"6cbf144d807c0594cd45c7126365e32e9e93e50f9db5e16f942dd2eaa3c9ff70"}
Jul 18 01:26:19.411046 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:19.411042    2554 scope.go:117] "RemoveContainer" containerID="e0cc5cd1c556d0ba3837d92289c1b2bf0d7e879d6ff755cfabe00ff0216d5a3a"
Jul 18 01:26:19.411635 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:19.411316    2554 scope.go:117] "RemoveContainer" containerID="6cbf144d807c0594cd45c7126365e32e9e93e50f9db5e16f942dd2eaa3c9ff70"
Jul 18 01:26:19.411635 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:26:19.411579    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk_odh-ai-gateway-infra(596ed215-8237-474a-afe7-703b817308b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" podUID="596ed215-8237-474a-afe7-703b817308b0"
Jul 18 01:26:20.417597 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:20.417568    2554 scope.go:117] "RemoveContainer" containerID="6cbf144d807c0594cd45c7126365e32e9e93e50f9db5e16f942dd2eaa3c9ff70"
Jul 18 01:26:20.417992 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:26:20.417822    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk_odh-ai-gateway-infra(596ed215-8237-474a-afe7-703b817308b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" podUID="596ed215-8237-474a-afe7-703b817308b0"
Jul 18 01:26:21.922810 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:21.922772    2554 patch_prober.go:28] interesting pod/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 01:26:21.923206 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:21.922836    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 01:26:22.971280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:22.971252    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:26:23.083746 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083710    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-data\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.083952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083756    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gknfp\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-kube-api-access-gknfp\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.083952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083778    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-envoy\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.083952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083835    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577a5dd3-8525-4e33-a403-03d47b653f91-istio-podinfo\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.083952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083887    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-certs\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.083952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083918    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577a5dd3-8525-4e33-a403-03d47b653f91-istiod-ca-cert\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.084233 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083970    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-socket\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.084233 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.083995    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-istio-token\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.084233 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.084027    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-credential-socket\") pod \"577a5dd3-8525-4e33-a403-03d47b653f91\" (UID: \"577a5dd3-8525-4e33-a403-03d47b653f91\") "
Jul 18 01:26:23.084401 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.084332    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:26:23.084609 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.084485    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.084609 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.084505    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:26:23.084609 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.084552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:26:23.084922 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.084729    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-data" (OuterVolumeSpecName: "istio-data") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:26:23.084922 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.084898    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577a5dd3-8525-4e33-a403-03d47b653f91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:26:23.086416 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.086389    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-istio-token" (OuterVolumeSpecName: "istio-token") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:26:23.086644 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.086596    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:26:23.086725 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.086655    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/577a5dd3-8525-4e33-a403-03d47b653f91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:26:23.087086 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.087054    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-kube-api-access-gknfp" (OuterVolumeSpecName: "kube-api-access-gknfp") pod "577a5dd3-8525-4e33-a403-03d47b653f91" (UID: "577a5dd3-8525-4e33-a403-03d47b653f91"). InnerVolumeSpecName "kube-api-access-gknfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:26:23.185191 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185150    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.185191 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185188    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gknfp\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-kube-api-access-gknfp\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.185191 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185198    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.185438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185207    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577a5dd3-8525-4e33-a403-03d47b653f91-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.185438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185216    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.185438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185224    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577a5dd3-8525-4e33-a403-03d47b653f91-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.185438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185231    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577a5dd3-8525-4e33-a403-03d47b653f91-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.185438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.185238    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577a5dd3-8525-4e33-a403-03d47b653f91-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:23.432898 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.432779    2554 generic.go:356] "Generic (PLEG): container finished" podID="577a5dd3-8525-4e33-a403-03d47b653f91" containerID="e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214" exitCode=0
Jul 18 01:26:23.432898 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.432844    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"
Jul 18 01:26:23.433135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.432954    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" event={"ID":"577a5dd3-8525-4e33-a403-03d47b653f91","Type":"ContainerDied","Data":"e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214"}
Jul 18 01:26:23.433135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.433006    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p" event={"ID":"577a5dd3-8525-4e33-a403-03d47b653f91","Type":"ContainerDied","Data":"7d289f819d88e047761fd1543c1a589e57fbf8b491f0077dd33a5e9ffd7f1f74"}
Jul 18 01:26:23.433135 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.433037    2554 scope.go:117] "RemoveContainer" containerID="e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214"
Jul 18 01:26:23.447841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.447783    2554 scope.go:117] "RemoveContainer" containerID="9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093"
Jul 18 01:26:23.458420 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.458394    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"]
Jul 18 01:26:23.459393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.459367    2554 scope.go:117] "RemoveContainer" containerID="e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214"
Jul 18 01:26:23.460211 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:26:23.460186    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214\": container with ID starting with e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214 not found: ID does not exist" containerID="e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214"
Jul 18 01:26:23.460331 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.460218    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214"} err="failed to get container status \"e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214\": rpc error: code = NotFound desc = could not find container \"e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214\": container with ID starting with e27d73409b79eb9591d48d32f050752bd0a385c1fb7c6584ef7a3e7477164214 not found: ID does not exist"
Jul 18 01:26:23.460331 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.460238    2554 scope.go:117] "RemoveContainer" containerID="9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093"
Jul 18 01:26:23.460538 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:26:23.460513    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093\": container with ID starting with 9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093 not found: ID does not exist" containerID="9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093"
Jul 18 01:26:23.460602 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.460546    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093"} err="failed to get container status \"9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093\": rpc error: code = NotFound desc = could not find container \"9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093\": container with ID starting with 9a1b90b53ef927327f21fe42ad1c25440cfc6f186991e066d840abacec509093 not found: ID does not exist"
Jul 18 01:26:23.462199 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:23.462179    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-46dd73e1-openshift-default-7c7784f5cf-d559p"]
Jul 18 01:26:24.336791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:24.336756    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" path="/var/lib/kubelet/pods/577a5dd3-8525-4e33-a403-03d47b653f91/volumes"
Jul 18 01:26:35.330692 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:35.330649    2554 scope.go:117] "RemoveContainer" containerID="6cbf144d807c0594cd45c7126365e32e9e93e50f9db5e16f942dd2eaa3c9ff70"
Jul 18 01:26:36.499729 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:36.499693    2554 generic.go:356] "Generic (PLEG): container finished" podID="596ed215-8237-474a-afe7-703b817308b0" containerID="c5f03688e03826e5b241ede6b1aa1f4176821ee3c33cc012b158181e831bd356" exitCode=6
Jul 18 01:26:36.500218 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:36.499749    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" event={"ID":"596ed215-8237-474a-afe7-703b817308b0","Type":"ContainerDied","Data":"c5f03688e03826e5b241ede6b1aa1f4176821ee3c33cc012b158181e831bd356"}
Jul 18 01:26:36.500218 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:36.499798    2554 scope.go:117] "RemoveContainer" containerID="6cbf144d807c0594cd45c7126365e32e9e93e50f9db5e16f942dd2eaa3c9ff70"
Jul 18 01:26:36.500218 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:36.500146    2554 scope.go:117] "RemoveContainer" containerID="c5f03688e03826e5b241ede6b1aa1f4176821ee3c33cc012b158181e831bd356"
Jul 18 01:26:36.500432 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:26:36.500409    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk_odh-ai-gateway-infra(596ed215-8237-474a-afe7-703b817308b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" podUID="596ed215-8237-474a-afe7-703b817308b0"
Jul 18 01:26:37.525093 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:37.525065    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"]
Jul 18 01:26:37.659411 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:37.659386    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"
Jul 18 01:26:37.732587 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:37.732551    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596ed215-8237-474a-afe7-703b817308b0-maas-api-service-ca\") pod \"596ed215-8237-474a-afe7-703b817308b0\" (UID: \"596ed215-8237-474a-afe7-703b817308b0\") "
Jul 18 01:26:37.732956 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:37.732934    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596ed215-8237-474a-afe7-703b817308b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "596ed215-8237-474a-afe7-703b817308b0" (UID: "596ed215-8237-474a-afe7-703b817308b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:26:37.834088 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:37.833992    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596ed215-8237-474a-afe7-703b817308b0-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:26:38.511703 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:38.511675    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"
Jul 18 01:26:38.511877 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:38.511674    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk" event={"ID":"596ed215-8237-474a-afe7-703b817308b0","Type":"ContainerDied","Data":"334968ad6fa90655489adb51ebe340b945a5dca44d98e126a360e397408ea5f3"}
Jul 18 01:26:38.511877 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:38.511806    2554 scope.go:117] "RemoveContainer" containerID="c5f03688e03826e5b241ede6b1aa1f4176821ee3c33cc012b158181e831bd356"
Jul 18 01:26:38.527889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:38.527862    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"]
Jul 18 01:26:38.532914 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:38.532891    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-m4pfk"]
Jul 18 01:26:40.335990 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:40.335959    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="596ed215-8237-474a-afe7-703b817308b0" path="/var/lib/kubelet/pods/596ed215-8237-474a-afe7-703b817308b0/volumes"
Jul 18 01:26:52.206254 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206175    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"]
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206564    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206576    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206585    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206591    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206597    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206602    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206625    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206631    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206641    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy"
Jul 18 01:26:52.206647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206646    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy"
Jul 18 01:26:52.207020 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206701    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.207020 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206710    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy"
Jul 18 01:26:52.207020 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206717    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="577a5dd3-8525-4e33-a403-03d47b653f91" containerName="istio-proxy"
Jul 18 01:26:52.207020 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206722    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd53f45c-dffa-4ec1-be2e-c91a4a638427" containerName="revoke-keys"
Jul 18 01:26:52.207020 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.206729    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:26:52.211160 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.211142    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"
Jul 18 01:26:52.213768 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.213746    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:26:52.217878 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.217841    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"]
Jul 18 01:26:52.280095 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.280063    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d33dae-cf8f-435f-9475-4231122eb0d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2\" (UID: \"48d33dae-cf8f-435f-9475-4231122eb0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"
Jul 18 01:26:52.380643 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.380582    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d33dae-cf8f-435f-9475-4231122eb0d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2\" (UID: \"48d33dae-cf8f-435f-9475-4231122eb0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"
Jul 18 01:26:52.381177 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.381155    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d33dae-cf8f-435f-9475-4231122eb0d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2\" (UID: \"48d33dae-cf8f-435f-9475-4231122eb0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"
Jul 18 01:26:52.522591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.522560    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"
Jul 18 01:26:52.869061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:52.868974    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"]
Jul 18 01:26:52.871971 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:26:52.871945    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d33dae_cf8f_435f_9475_4231122eb0d8.slice/crio-60d8b6f1b46b2f911e384e7d3e59b300a548037392016d83f3fb6bdcc9a7d8c1 WatchSource:0}: Error finding container 60d8b6f1b46b2f911e384e7d3e59b300a548037392016d83f3fb6bdcc9a7d8c1: Status 404 returned error can't find the container with id 60d8b6f1b46b2f911e384e7d3e59b300a548037392016d83f3fb6bdcc9a7d8c1
Jul 18 01:26:53.589506 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:53.589466    2554 generic.go:356] "Generic (PLEG): container finished" podID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerID="710624aacaa9a74a210557b82c900ca1f349f404edda73e3f61183b5c82bbd8f" exitCode=6
Jul 18 01:26:53.589931 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:53.589511    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" event={"ID":"48d33dae-cf8f-435f-9475-4231122eb0d8","Type":"ContainerDied","Data":"710624aacaa9a74a210557b82c900ca1f349f404edda73e3f61183b5c82bbd8f"}
Jul 18 01:26:53.589931 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:53.589585    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" event={"ID":"48d33dae-cf8f-435f-9475-4231122eb0d8","Type":"ContainerStarted","Data":"60d8b6f1b46b2f911e384e7d3e59b300a548037392016d83f3fb6bdcc9a7d8c1"}
Jul 18 01:26:53.589931 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:53.589821    2554 scope.go:117] "RemoveContainer" containerID="710624aacaa9a74a210557b82c900ca1f349f404edda73e3f61183b5c82bbd8f"
Jul 18 01:26:54.596451 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:54.596411    2554 generic.go:356] "Generic (PLEG): container finished" podID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerID="23e2b42bf8ba4f3680309562c2b673c96fe92f61a7c2bd4983ec2064d3f1d3cc" exitCode=6
Jul 18 01:26:54.596930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:54.596500    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" event={"ID":"48d33dae-cf8f-435f-9475-4231122eb0d8","Type":"ContainerDied","Data":"23e2b42bf8ba4f3680309562c2b673c96fe92f61a7c2bd4983ec2064d3f1d3cc"}
Jul 18 01:26:54.596930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:54.596576    2554 scope.go:117] "RemoveContainer" containerID="710624aacaa9a74a210557b82c900ca1f349f404edda73e3f61183b5c82bbd8f"
Jul 18 01:26:54.596930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:54.596821    2554 scope.go:117] "RemoveContainer" containerID="23e2b42bf8ba4f3680309562c2b673c96fe92f61a7c2bd4983ec2064d3f1d3cc"
Jul 18 01:26:54.597107 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:26:54.597086    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2_odh-ai-gateway-infra(48d33dae-cf8f-435f-9475-4231122eb0d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8"
Jul 18 01:26:55.603096 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:26:55.603067    2554 scope.go:117] "RemoveContainer" containerID="23e2b42bf8ba4f3680309562c2b673c96fe92f61a7c2bd4983ec2064d3f1d3cc"
Jul 18 01:26:55.603501 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:26:55.603252    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2_odh-ai-gateway-infra(48d33dae-cf8f-435f-9475-4231122eb0d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8"
Jul 18 01:27:09.330511 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:09.330480    2554 scope.go:117] "RemoveContainer" containerID="23e2b42bf8ba4f3680309562c2b673c96fe92f61a7c2bd4983ec2064d3f1d3cc"
Jul 18 01:27:09.674259 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:09.674224    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" event={"ID":"48d33dae-cf8f-435f-9475-4231122eb0d8","Type":"ContainerStarted","Data":"55a9fc7063be3b0ca52746ea632610a38aa83db416869e80226efb616aee6a19"}
Jul 18 01:27:09.691928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:09.691880    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" podStartSLOduration=17.691863196 podStartE2EDuration="17.691863196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:09.690638689 +0000 UTC m=+2785.816573658" watchObservedRunningTime="2026-07-18 01:27:09.691863196 +0000 UTC m=+2785.817798174"
Jul 18 01:27:10.680268 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:10.680231    2554 generic.go:356] "Generic (PLEG): container finished" podID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerID="55a9fc7063be3b0ca52746ea632610a38aa83db416869e80226efb616aee6a19" exitCode=6
Jul 18 01:27:10.680766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:10.680303    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" event={"ID":"48d33dae-cf8f-435f-9475-4231122eb0d8","Type":"ContainerDied","Data":"55a9fc7063be3b0ca52746ea632610a38aa83db416869e80226efb616aee6a19"}
Jul 18 01:27:10.680766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:10.680364    2554 scope.go:117] "RemoveContainer" containerID="23e2b42bf8ba4f3680309562c2b673c96fe92f61a7c2bd4983ec2064d3f1d3cc"
Jul 18 01:27:10.680766 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:10.680677    2554 scope.go:117] "RemoveContainer" containerID="55a9fc7063be3b0ca52746ea632610a38aa83db416869e80226efb616aee6a19"
Jul 18 01:27:10.680978 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:27:10.680959    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2_odh-ai-gateway-infra(48d33dae-cf8f-435f-9475-4231122eb0d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8"
Jul 18 01:27:10.701111 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:10.701072    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"]
Jul 18 01:27:11.593490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.593457    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"]
Jul 18 01:27:11.594159 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.594133    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="596ed215-8237-474a-afe7-703b817308b0" containerName="revoke-keys"
Jul 18 01:27:11.598992 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.598971    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.601634 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.601595    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-390947-openshift-default-dockercfg-2tf5c\""
Jul 18 01:27:11.610986 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.610960    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"]
Jul 18 01:27:11.678320 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678271    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-credential-socket\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678515 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c229a0-1500-4637-b70e-fdd83aa0ed41-istiod-ca-cert\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678515 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-socket\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678515 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678496    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-data\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678553    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfv6\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-kube-api-access-dwfv6\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678588    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-certs\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678610    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-token\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678695    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-envoy\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.678727 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.678721    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-podinfo\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779246 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-data\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779275    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfv6\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-kube-api-access-dwfv6\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-certs\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779350    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-token\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779402    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-envoy\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779432    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-podinfo\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779497    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-credential-socket\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779530    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c229a0-1500-4637-b70e-fdd83aa0ed41-istiod-ca-cert\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.779717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779573    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-socket\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.780131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.779723    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-data\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.780131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.780040    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-certs\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.780131 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.780119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-socket\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.780479 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.780451    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-credential-socket\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.780571 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.780549    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c229a0-1500-4637-b70e-fdd83aa0ed41-istiod-ca-cert\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.782377 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.782350    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-envoy\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.782522 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.782503    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-podinfo\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.791708 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.791681    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-token\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.791873 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.791858    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfv6\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-kube-api-access-dwfv6\") pod \"e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") " pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.832196 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.832169    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"
Jul 18 01:27:11.880965 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.880841    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d33dae-cf8f-435f-9475-4231122eb0d8-maas-api-service-ca\") pod \"48d33dae-cf8f-435f-9475-4231122eb0d8\" (UID: \"48d33dae-cf8f-435f-9475-4231122eb0d8\") "
Jul 18 01:27:11.881249 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.881218    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d33dae-cf8f-435f-9475-4231122eb0d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48d33dae-cf8f-435f-9475-4231122eb0d8" (UID: "48d33dae-cf8f-435f-9475-4231122eb0d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:27:11.911969 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.911938    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:11.982631 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:11.982589    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d33dae-cf8f-435f-9475-4231122eb0d8-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:27:12.047912 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.047883    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"]
Jul 18 01:27:12.051226 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:27:12.051190    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c229a0_1500_4637_b70e_fdd83aa0ed41.slice/crio-0756782440bba519a78b582aa809654caa175dc8212486f30a98753200752a86 WatchSource:0}: Error finding container 0756782440bba519a78b582aa809654caa175dc8212486f30a98753200752a86: Status 404 returned error can't find the container with id 0756782440bba519a78b582aa809654caa175dc8212486f30a98753200752a86
Jul 18 01:27:12.053479 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.053445    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:12.053565 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.053513    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:12.053565 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.053545    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:12.695828 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.695743    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"
Jul 18 01:27:12.695828 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.695735    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2" event={"ID":"48d33dae-cf8f-435f-9475-4231122eb0d8","Type":"ContainerDied","Data":"60d8b6f1b46b2f911e384e7d3e59b300a548037392016d83f3fb6bdcc9a7d8c1"}
Jul 18 01:27:12.696060 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.695841    2554 scope.go:117] "RemoveContainer" containerID="55a9fc7063be3b0ca52746ea632610a38aa83db416869e80226efb616aee6a19"
Jul 18 01:27:12.697459 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.697416    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" event={"ID":"92c229a0-1500-4637-b70e-fdd83aa0ed41","Type":"ContainerStarted","Data":"5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a"}
Jul 18 01:27:12.697568 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.697487    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" event={"ID":"92c229a0-1500-4637-b70e-fdd83aa0ed41","Type":"ContainerStarted","Data":"0756782440bba519a78b582aa809654caa175dc8212486f30a98753200752a86"}
Jul 18 01:27:12.712829 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.712801    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"]
Jul 18 01:27:12.718193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.718170    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-k97m2"]
Jul 18 01:27:12.735829 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.735780    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" podStartSLOduration=1.73576298 podStartE2EDuration="1.73576298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:12.735034329 +0000 UTC m=+2788.860969310" watchObservedRunningTime="2026-07-18 01:27:12.73576298 +0000 UTC m=+2788.861698033"
Jul 18 01:27:12.912971 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:12.912936    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:13.918173 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:13.918143    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:14.337417 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:14.337383    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" path="/var/lib/kubelet/pods/48d33dae-cf8f-435f-9475-4231122eb0d8/volumes"
Jul 18 01:27:14.709137 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:14.709046    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:14.710114 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:14.710086    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:15.147712 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.147676    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"]
Jul 18 01:27:15.148105 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148081    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148105 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148094    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148105 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148104    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148214 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148109    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148214 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148118    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148214 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148124    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148214 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148189    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148214 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148197    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.148214 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.148204    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d33dae-cf8f-435f-9475-4231122eb0d8" containerName="revoke-keys"
Jul 18 01:27:15.152740 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.152723    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"
Jul 18 01:27:15.155463 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.155437    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:27:15.158356 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.158325    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"]
Jul 18 01:27:15.214888 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.214847    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1334e7a9-13f4-4d97-a0d1-17a9e2295bb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-9q25t\" (UID: \"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"
Jul 18 01:27:15.316360 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.316316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1334e7a9-13f4-4d97-a0d1-17a9e2295bb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-9q25t\" (UID: \"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"
Jul 18 01:27:15.316965 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.316947    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1334e7a9-13f4-4d97-a0d1-17a9e2295bb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-9q25t\" (UID: \"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"
Jul 18 01:27:15.465982 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.465886    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"
Jul 18 01:27:15.617272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.617245    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"]
Jul 18 01:27:15.618939 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:27:15.618904    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1334e7a9_13f4_4d97_a0d1_17a9e2295bb2.slice/crio-5534b01d90d049e05cc9786f220c21d5db37202fa2525fdacc36ebeedd150ad6 WatchSource:0}: Error finding container 5534b01d90d049e05cc9786f220c21d5db37202fa2525fdacc36ebeedd150ad6: Status 404 returned error can't find the container with id 5534b01d90d049e05cc9786f220c21d5db37202fa2525fdacc36ebeedd150ad6
Jul 18 01:27:15.714359 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:15.714324    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" event={"ID":"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2","Type":"ContainerStarted","Data":"5534b01d90d049e05cc9786f220c21d5db37202fa2525fdacc36ebeedd150ad6"}
Jul 18 01:27:16.720231 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:16.720144    2554 generic.go:356] "Generic (PLEG): container finished" podID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerID="ed3debbd59f079d895d0bcd3be71af6bd29b700da3aba135bf2f10d006c4c4f2" exitCode=6
Jul 18 01:27:16.720611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:16.720236    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" event={"ID":"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2","Type":"ContainerDied","Data":"ed3debbd59f079d895d0bcd3be71af6bd29b700da3aba135bf2f10d006c4c4f2"}
Jul 18 01:27:16.720611 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:16.720429    2554 scope.go:117] "RemoveContainer" containerID="ed3debbd59f079d895d0bcd3be71af6bd29b700da3aba135bf2f10d006c4c4f2"
Jul 18 01:27:17.154285 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.154247    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"]
Jul 18 01:27:17.157898 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.157874    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"
Jul 18 01:27:17.163891 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.163811    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"]
Jul 18 01:27:17.236306 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.236264    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d7dfd-4026-4bd4-853e-77394dc2eb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb\" (UID: \"426d7dfd-4026-4bd4-853e-77394dc2eb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"
Jul 18 01:27:17.337726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.337693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d7dfd-4026-4bd4-853e-77394dc2eb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb\" (UID: \"426d7dfd-4026-4bd4-853e-77394dc2eb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"
Jul 18 01:27:17.338324 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.338303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d7dfd-4026-4bd4-853e-77394dc2eb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb\" (UID: \"426d7dfd-4026-4bd4-853e-77394dc2eb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"
Jul 18 01:27:17.470515 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.470477    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"
Jul 18 01:27:17.607305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.607279    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"]
Jul 18 01:27:17.610229 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:27:17.610199    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d7dfd_4026_4bd4_853e_77394dc2eb84.slice/crio-22fa42c45b2a7e01a03f8a2f3fd919b4857604ca7249a76c27b079a39f486ed0 WatchSource:0}: Error finding container 22fa42c45b2a7e01a03f8a2f3fd919b4857604ca7249a76c27b079a39f486ed0: Status 404 returned error can't find the container with id 22fa42c45b2a7e01a03f8a2f3fd919b4857604ca7249a76c27b079a39f486ed0
Jul 18 01:27:17.726812 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.726782    2554 generic.go:356] "Generic (PLEG): container finished" podID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerID="3f550766eb82f1db733dee840c7b39aae4b314a0c6c9eb8f80e328ce1bb8ec0e" exitCode=6
Jul 18 01:27:17.727246 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.726869    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" event={"ID":"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2","Type":"ContainerDied","Data":"3f550766eb82f1db733dee840c7b39aae4b314a0c6c9eb8f80e328ce1bb8ec0e"}
Jul 18 01:27:17.727246 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.726938    2554 scope.go:117] "RemoveContainer" containerID="ed3debbd59f079d895d0bcd3be71af6bd29b700da3aba135bf2f10d006c4c4f2"
Jul 18 01:27:17.727246 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.727105    2554 scope.go:117] "RemoveContainer" containerID="3f550766eb82f1db733dee840c7b39aae4b314a0c6c9eb8f80e328ce1bb8ec0e"
Jul 18 01:27:17.727406 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:27:17.727377    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-9q25t_odh-ai-gateway-infra(1334e7a9-13f4-4d97-a0d1-17a9e2295bb2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2"
Jul 18 01:27:17.728193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:17.728172    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" event={"ID":"426d7dfd-4026-4bd4-853e-77394dc2eb84","Type":"ContainerStarted","Data":"22fa42c45b2a7e01a03f8a2f3fd919b4857604ca7249a76c27b079a39f486ed0"}
Jul 18 01:27:18.735003 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:18.734968    2554 generic.go:356] "Generic (PLEG): container finished" podID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerID="2d3fd653290f959096f27a2a3a24b67de54d949d9eb1488b592e51b3f7dd3294" exitCode=6
Jul 18 01:27:18.735435 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:18.735059    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" event={"ID":"426d7dfd-4026-4bd4-853e-77394dc2eb84","Type":"ContainerDied","Data":"2d3fd653290f959096f27a2a3a24b67de54d949d9eb1488b592e51b3f7dd3294"}
Jul 18 01:27:18.735435 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:18.735296    2554 scope.go:117] "RemoveContainer" containerID="2d3fd653290f959096f27a2a3a24b67de54d949d9eb1488b592e51b3f7dd3294"
Jul 18 01:27:18.736944 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:18.736927    2554 scope.go:117] "RemoveContainer" containerID="3f550766eb82f1db733dee840c7b39aae4b314a0c6c9eb8f80e328ce1bb8ec0e"
Jul 18 01:27:18.737125 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:27:18.737109    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-9q25t_odh-ai-gateway-infra(1334e7a9-13f4-4d97-a0d1-17a9e2295bb2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2"
Jul 18 01:27:19.729775 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.729689    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"]
Jul 18 01:27:19.733552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.733530    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.736056 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.736033    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a3406-openshift-default-dockercfg-986wg\""
Jul 18 01:27:19.744441 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.744415    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"]
Jul 18 01:27:19.744982 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.744952    2554 generic.go:356] "Generic (PLEG): container finished" podID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerID="1a7b3a3df85683b8c74c629bf0567343e69c33eb7dde027d8e84a0d89a158f58" exitCode=6
Jul 18 01:27:19.745149 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.745037    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" event={"ID":"426d7dfd-4026-4bd4-853e-77394dc2eb84","Type":"ContainerDied","Data":"1a7b3a3df85683b8c74c629bf0567343e69c33eb7dde027d8e84a0d89a158f58"}
Jul 18 01:27:19.745318 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.745297    2554 scope.go:117] "RemoveContainer" containerID="2d3fd653290f959096f27a2a3a24b67de54d949d9eb1488b592e51b3f7dd3294"
Jul 18 01:27:19.745679 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.745658    2554 scope.go:117] "RemoveContainer" containerID="1a7b3a3df85683b8c74c629bf0567343e69c33eb7dde027d8e84a0d89a158f58"
Jul 18 01:27:19.746015 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:27:19.745985    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb_odh-ai-gateway-infra(426d7dfd-4026-4bd4-853e-77394dc2eb84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84"
Jul 18 01:27:19.866394 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866356    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-envoy\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866401    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnvj\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-kube-api-access-lxnvj\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866433    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-certs\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866496    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-podinfo\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866586 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866552    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-credential-socket\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866588    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-data\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istiod-ca-cert\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866775    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-socket\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.866830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.866795    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-token\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.967512 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967460    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-podinfo\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.967742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967544    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-credential-socket\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.967742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-data\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.967742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istiod-ca-cert\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.967742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-socket\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.967742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-token\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968034 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967769    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-envoy\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968034 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967810    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnvj\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-kube-api-access-lxnvj\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968034 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.967848    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-certs\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.968027    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-credential-socket\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.968064    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-data\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968304 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.968272    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-socket\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968357 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.968301    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-certs\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.968431 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.968412    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istiod-ca-cert\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.970010 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.969990    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-podinfo\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.970103 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.970030    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-envoy\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.977762 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.977740    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnvj\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-kube-api-access-lxnvj\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:19.977762 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:19.977760    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-token\") pod \"e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") " pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:20.047745 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.047713    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:20.185543 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.185364    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"]
Jul 18 01:27:20.188344 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:27:20.188311    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9c4ce5_94d1_4d43_bcd3_15e92cd7a90a.slice/crio-3b0adbc202d7c9724cbc8127924f171fac3cf83e9ea6f3c9c40d467ad688d5aa WatchSource:0}: Error finding container 3b0adbc202d7c9724cbc8127924f171fac3cf83e9ea6f3c9c40d467ad688d5aa: Status 404 returned error can't find the container with id 3b0adbc202d7c9724cbc8127924f171fac3cf83e9ea6f3c9c40d467ad688d5aa
Jul 18 01:27:20.190519 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.190490    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:20.190576 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.190556    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:20.190636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.190582    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:20.751587 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.751548    2554 scope.go:117] "RemoveContainer" containerID="1a7b3a3df85683b8c74c629bf0567343e69c33eb7dde027d8e84a0d89a158f58"
Jul 18 01:27:20.752003 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:27:20.751849    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb_odh-ai-gateway-infra(426d7dfd-4026-4bd4-853e-77394dc2eb84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84"
Jul 18 01:27:20.752842 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.752817    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" event={"ID":"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a","Type":"ContainerStarted","Data":"97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee"}
Jul 18 01:27:20.752963 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.752869    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" event={"ID":"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a","Type":"ContainerStarted","Data":"3b0adbc202d7c9724cbc8127924f171fac3cf83e9ea6f3c9c40d467ad688d5aa"}
Jul 18 01:27:20.786073 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:20.786021    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" podStartSLOduration=1.786006575 podStartE2EDuration="1.786006575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:20.783911168 +0000 UTC m=+2796.909846160" watchObservedRunningTime="2026-07-18 01:27:20.786006575 +0000 UTC m=+2796.911941554"
Jul 18 01:27:21.048217 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:21.048132    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:21.053415 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:21.053390    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:21.758352 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:21.758311    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:21.759231 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:21.759210    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:30.749842 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:30.749791    2554 patch_prober.go:28] interesting pod/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 18 01:27:30.750442 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:30.749861    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 18 01:27:31.835367 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:31.835334    2554 generic.go:356] "Generic (PLEG): container finished" podID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerID="5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a" exitCode=0
Jul 18 01:27:31.835842 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:31.835403    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" event={"ID":"92c229a0-1500-4637-b70e-fdd83aa0ed41","Type":"ContainerDied","Data":"5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a"}
Jul 18 01:27:31.835931 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:31.835913    2554 scope.go:117] "RemoveContainer" containerID="5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a"
Jul 18 01:27:31.837061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:31.837032    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:31.837134 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:31.837106    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:31.837174 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:31.837140    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:32.331162 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:32.331131    2554 scope.go:117] "RemoveContainer" containerID="1a7b3a3df85683b8c74c629bf0567343e69c33eb7dde027d8e84a0d89a158f58"
Jul 18 01:27:32.847791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:32.847075    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" event={"ID":"426d7dfd-4026-4bd4-853e-77394dc2eb84","Type":"ContainerStarted","Data":"c90bb943d5b845c88243b40d0b5736acd79bbbff1ce6ce862b0a8a1f1de46fc3"}
Jul 18 01:27:32.850806 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:32.850081    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" event={"ID":"92c229a0-1500-4637-b70e-fdd83aa0ed41","Type":"ContainerStarted","Data":"60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623"}
Jul 18 01:27:32.864503 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:32.864452    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" podStartSLOduration=15.864435154 podStartE2EDuration="15.864435154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:32.863371107 +0000 UTC m=+2808.989306086" watchObservedRunningTime="2026-07-18 01:27:32.864435154 +0000 UTC m=+2808.990370133"
Jul 18 01:27:32.912378 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:32.912271    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:32.912761 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:32.912729    2554 patch_prober.go:28] interesting pod/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Jul 18 01:27:32.912889 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:32.912793    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Jul 18 01:27:33.330772 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.330731    2554 scope.go:117] "RemoveContainer" containerID="3f550766eb82f1db733dee840c7b39aae4b314a0c6c9eb8f80e328ce1bb8ec0e"
Jul 18 01:27:33.859992 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.859892    2554 generic.go:356] "Generic (PLEG): container finished" podID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerID="c90bb943d5b845c88243b40d0b5736acd79bbbff1ce6ce862b0a8a1f1de46fc3" exitCode=6
Jul 18 01:27:33.860472 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.860045    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" event={"ID":"426d7dfd-4026-4bd4-853e-77394dc2eb84","Type":"ContainerDied","Data":"c90bb943d5b845c88243b40d0b5736acd79bbbff1ce6ce862b0a8a1f1de46fc3"}
Jul 18 01:27:33.860472 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.860106    2554 scope.go:117] "RemoveContainer" containerID="1a7b3a3df85683b8c74c629bf0567343e69c33eb7dde027d8e84a0d89a158f58"
Jul 18 01:27:33.860600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.860506    2554 scope.go:117] "RemoveContainer" containerID="c90bb943d5b845c88243b40d0b5736acd79bbbff1ce6ce862b0a8a1f1de46fc3"
Jul 18 01:27:33.860818 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:27:33.860793    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb_odh-ai-gateway-infra(426d7dfd-4026-4bd4-853e-77394dc2eb84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84"
Jul 18 01:27:33.863698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.863671    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" event={"ID":"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2","Type":"ContainerStarted","Data":"b77d2b1569ab1213dd4741bdff3ddbec668b152c5773967124113fe01e9857ad"}
Jul 18 01:27:33.875117 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.875094    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"]
Jul 18 01:27:33.901407 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.901358    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" podStartSLOduration=18.901332489 podStartE2EDuration="18.901332489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:33.900687017 +0000 UTC m=+2810.026621993" watchObservedRunningTime="2026-07-18 01:27:33.901332489 +0000 UTC m=+2810.027267469"
Jul 18 01:27:33.913203 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.913160    2554 patch_prober.go:28] interesting pod/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Jul 18 01:27:33.913387 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:33.913227    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Jul 18 01:27:34.870332 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:34.870297    2554 generic.go:356] "Generic (PLEG): container finished" podID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerID="b77d2b1569ab1213dd4741bdff3ddbec668b152c5773967124113fe01e9857ad" exitCode=6
Jul 18 01:27:34.870798 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:34.870384    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" event={"ID":"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2","Type":"ContainerDied","Data":"b77d2b1569ab1213dd4741bdff3ddbec668b152c5773967124113fe01e9857ad"}
Jul 18 01:27:34.870798 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:34.870455    2554 scope.go:117] "RemoveContainer" containerID="3f550766eb82f1db733dee840c7b39aae4b314a0c6c9eb8f80e328ce1bb8ec0e"
Jul 18 01:27:34.870917 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:34.870820    2554 scope.go:117] "RemoveContainer" containerID="b77d2b1569ab1213dd4741bdff3ddbec668b152c5773967124113fe01e9857ad"
Jul 18 01:27:34.871117 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:27:34.871091    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-9q25t_odh-ai-gateway-infra(1334e7a9-13f4-4d97-a0d1-17a9e2295bb2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2"
Jul 18 01:27:34.925637 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:34.924675    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"]
Jul 18 01:27:34.931639 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:34.931033    2554 patch_prober.go:28] interesting pod/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Jul 18 01:27:34.931639 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:34.931099    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Jul 18 01:27:35.062756 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.062732    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"
Jul 18 01:27:35.249235 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.249196    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d7dfd-4026-4bd4-853e-77394dc2eb84-maas-api-service-ca\") pod \"426d7dfd-4026-4bd4-853e-77394dc2eb84\" (UID: \"426d7dfd-4026-4bd4-853e-77394dc2eb84\") "
Jul 18 01:27:35.249561 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.249529    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426d7dfd-4026-4bd4-853e-77394dc2eb84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "426d7dfd-4026-4bd4-853e-77394dc2eb84" (UID: "426d7dfd-4026-4bd4-853e-77394dc2eb84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:27:35.249742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.249723    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d7dfd-4026-4bd4-853e-77394dc2eb84-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:27:35.881659 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.881627    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"
Jul 18 01:27:35.881659 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.881636    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb" event={"ID":"426d7dfd-4026-4bd4-853e-77394dc2eb84","Type":"ContainerDied","Data":"22fa42c45b2a7e01a03f8a2f3fd919b4857604ca7249a76c27b079a39f486ed0"}
Jul 18 01:27:35.882101 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.881746    2554 scope.go:117] "RemoveContainer" containerID="c90bb943d5b845c88243b40d0b5736acd79bbbff1ce6ce862b0a8a1f1de46fc3"
Jul 18 01:27:35.913162 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.912710    2554 patch_prober.go:28] interesting pod/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Jul 18 01:27:35.913162 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.912775    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Jul 18 01:27:35.918006 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.917948    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"]
Jul 18 01:27:35.922689 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:35.922184    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bhbdb"]
Jul 18 01:27:36.068555 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.068530    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"
Jul 18 01:27:36.159220 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.159127    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1334e7a9-13f4-4d97-a0d1-17a9e2295bb2-maas-api-service-ca\") pod \"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2\" (UID: \"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2\") "
Jul 18 01:27:36.159503 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.159470    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1334e7a9-13f4-4d97-a0d1-17a9e2295bb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" (UID: "1334e7a9-13f4-4d97-a0d1-17a9e2295bb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:27:36.260114 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.260073    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1334e7a9-13f4-4d97-a0d1-17a9e2295bb2-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:27:36.339332 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.339296    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" path="/var/lib/kubelet/pods/426d7dfd-4026-4bd4-853e-77394dc2eb84/volumes"
Jul 18 01:27:36.889033 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.888998    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"
Jul 18 01:27:36.889526 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.889037    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t" event={"ID":"1334e7a9-13f4-4d97-a0d1-17a9e2295bb2","Type":"ContainerDied","Data":"5534b01d90d049e05cc9786f220c21d5db37202fa2525fdacc36ebeedd150ad6"}
Jul 18 01:27:36.889526 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.889104    2554 scope.go:117] "RemoveContainer" containerID="b77d2b1569ab1213dd4741bdff3ddbec668b152c5773967124113fe01e9857ad"
Jul 18 01:27:36.907034 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.907008    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"]
Jul 18 01:27:36.912350 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.912323    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-9q25t"]
Jul 18 01:27:36.915884 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.915862    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:36.916273 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.916250    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:36.916987 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:36.916969    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:27:37.800792 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:37.800745    2554 patch_prober.go:28] interesting pod/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 18 01:27:37.800991 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:37.800825    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 18 01:27:38.336596 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:38.336516    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" path="/var/lib/kubelet/pods/1334e7a9-13f4-4d97-a0d1-17a9e2295bb2/volumes"
Jul 18 01:27:38.902791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:38.902757    2554 generic.go:356] "Generic (PLEG): container finished" podID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerID="97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee" exitCode=0
Jul 18 01:27:38.902978 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:38.902831    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" event={"ID":"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a","Type":"ContainerDied","Data":"97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee"}
Jul 18 01:27:38.903369 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:38.903352    2554 scope.go:117] "RemoveContainer" containerID="97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee"
Jul 18 01:27:38.904380 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:38.904357    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:38.904435 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:38.904419    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:38.904496 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:38.904484    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:27:39.912272 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:39.912236    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" event={"ID":"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a","Type":"ContainerStarted","Data":"56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94"}
Jul 18 01:27:40.048119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:40.048078    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:40.048338 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:40.048315    2554 patch_prober.go:28] interesting pod/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Jul 18 01:27:40.048386 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:40.048369    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Jul 18 01:27:41.048311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:41.048270    2554 patch_prober.go:28] interesting pod/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Jul 18 01:27:41.048761 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:41.048344    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Jul 18 01:27:42.048419 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:42.048375    2554 patch_prober.go:28] interesting pod/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Jul 18 01:27:42.048864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:42.048453    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Jul 18 01:27:43.048303 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:43.048266    2554 patch_prober.go:28] interesting pod/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Jul 18 01:27:43.048490 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:43.048321    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Jul 18 01:27:44.051925 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:44.051895    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:44.052406 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:44.052275    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:27:44.053043 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:27:44.053021    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:28:57.486494 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:28:57.486458    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"]
Jul 18 01:28:57.487099 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:28:57.486723    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy" containerID="cri-o://60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623" gracePeriod=30
Jul 18 01:29:02.745204 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.745181    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:29:02.831120 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831041    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-podinfo\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831120 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831092    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-certs\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831162    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c229a0-1500-4637-b70e-fdd83aa0ed41-istiod-ca-cert\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831232    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-token\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831265    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-socket\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831305    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwfv6\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-kube-api-access-dwfv6\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831365 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831342    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-data\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831680 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831380    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-credential-socket\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831680 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831423    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-envoy\") pod \"92c229a0-1500-4637-b70e-fdd83aa0ed41\" (UID: \"92c229a0-1500-4637-b70e-fdd83aa0ed41\") "
Jul 18 01:29:02.831680 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831452    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:02.831680 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831554    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c229a0-1500-4637-b70e-fdd83aa0ed41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:02.831906 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831714    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:02.831906 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831889    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:02.832017 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831919    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.832017 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831939    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c229a0-1500-4637-b70e-fdd83aa0ed41-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.832017 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.831952    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.832336 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.832275    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-data" (OuterVolumeSpecName: "istio-data") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:02.833468 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.833443    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-token" (OuterVolumeSpecName: "istio-token") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:29:02.833584 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.833481    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:29:02.833954 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.833930    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:02.834045 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.833981    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-kube-api-access-dwfv6" (OuterVolumeSpecName: "kube-api-access-dwfv6") pod "92c229a0-1500-4637-b70e-fdd83aa0ed41" (UID: "92c229a0-1500-4637-b70e-fdd83aa0ed41"). InnerVolumeSpecName "kube-api-access-dwfv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:29:02.932776 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.932741    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.932776 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.932776    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.933011 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.932791    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.933011 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.932804    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.933011 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.932821    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwfv6\" (UniqueName: \"kubernetes.io/projected/92c229a0-1500-4637-b70e-fdd83aa0ed41-kube-api-access-dwfv6\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:02.933011 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:02.932836    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c229a0-1500-4637-b70e-fdd83aa0ed41-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:03.332600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.332564    2554 generic.go:356] "Generic (PLEG): container finished" podID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerID="60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623" exitCode=0
Jul 18 01:29:03.332814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.332639    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"
Jul 18 01:29:03.332814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.332658    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" event={"ID":"92c229a0-1500-4637-b70e-fdd83aa0ed41","Type":"ContainerDied","Data":"60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623"}
Jul 18 01:29:03.332814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.332713    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx" event={"ID":"92c229a0-1500-4637-b70e-fdd83aa0ed41","Type":"ContainerDied","Data":"0756782440bba519a78b582aa809654caa175dc8212486f30a98753200752a86"}
Jul 18 01:29:03.332814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.332744    2554 scope.go:117] "RemoveContainer" containerID="60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623"
Jul 18 01:29:03.342553 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.342534    2554 scope.go:117] "RemoveContainer" containerID="5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a"
Jul 18 01:29:03.350538 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.350515    2554 scope.go:117] "RemoveContainer" containerID="60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623"
Jul 18 01:29:03.350836 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:29:03.350816    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623\": container with ID starting with 60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623 not found: ID does not exist" containerID="60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623"
Jul 18 01:29:03.350913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.350846    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623"} err="failed to get container status \"60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623\": rpc error: code = NotFound desc = could not find container \"60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623\": container with ID starting with 60d3a3058ea9b5fd3c76cde27ad16b115f63789f2e84efb7ec50d8d741c4c623 not found: ID does not exist"
Jul 18 01:29:03.350913 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.350864    2554 scope.go:117] "RemoveContainer" containerID="5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a"
Jul 18 01:29:03.351134 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:29:03.351116    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a\": container with ID starting with 5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a not found: ID does not exist" containerID="5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a"
Jul 18 01:29:03.351176 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.351143    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a"} err="failed to get container status \"5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a\": rpc error: code = NotFound desc = could not find container \"5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a\": container with ID starting with 5e4d8d62fe9ab3b4fdc583d3b06e05fb5f7fda6c79580c6f0f2cb037b0893e2a not found: ID does not exist"
Jul 18 01:29:03.356029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.356004    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"]
Jul 18 01:29:03.360701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:03.360680    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-390947-openshift-default-d4f995fcc-j7ttx"]
Jul 18 01:29:04.337477 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:04.337433    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" path="/var/lib/kubelet/pods/92c229a0-1500-4637-b70e-fdd83aa0ed41/volumes"
Jul 18 01:29:19.196962 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.196919    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"]
Jul 18 01:29:19.197344 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197309    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197344 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197320    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197344 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197330    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy"
Jul 18 01:29:19.197344 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197335    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy"
Jul 18 01:29:19.197344 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197343    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197348    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197355    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197360    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197370    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197375    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197381    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197386    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197395    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197401    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197453    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197461    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197468    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197475    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197481    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197487    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197492    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1334e7a9-13f4-4d97-a0d1-17a9e2295bb2" containerName="revoke-keys"
Jul 18 01:29:19.197504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.197498    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c229a0-1500-4637-b70e-fdd83aa0ed41" containerName="istio-proxy"
Jul 18 01:29:19.202404 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.202380    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"
Jul 18 01:29:19.205058 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.205031    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:29:19.207652 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.207609    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"]
Jul 18 01:29:19.296101 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.296071    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f5e571-1e96-4c6c-a737-75c91e925261-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd\" (UID: \"c4f5e571-1e96-4c6c-a737-75c91e925261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"
Jul 18 01:29:19.397072 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.397031    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f5e571-1e96-4c6c-a737-75c91e925261-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd\" (UID: \"c4f5e571-1e96-4c6c-a737-75c91e925261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"
Jul 18 01:29:19.397698 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.397680    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f5e571-1e96-4c6c-a737-75c91e925261-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd\" (UID: \"c4f5e571-1e96-4c6c-a737-75c91e925261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"
Jul 18 01:29:19.514429 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.514400    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"
Jul 18 01:29:19.660346 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:19.660320    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"]
Jul 18 01:29:19.662533 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:29:19.662498    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f5e571_1e96_4c6c_a737_75c91e925261.slice/crio-08c61ea7aef9de363eba93eae3d2a8bb6bad2131fe1c46a9251739cf69857658 WatchSource:0}: Error finding container 08c61ea7aef9de363eba93eae3d2a8bb6bad2131fe1c46a9251739cf69857658: Status 404 returned error can't find the container with id 08c61ea7aef9de363eba93eae3d2a8bb6bad2131fe1c46a9251739cf69857658
Jul 18 01:29:20.422593 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:20.422547    2554 generic.go:356] "Generic (PLEG): container finished" podID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerID="d2586f5c3a5056e36081bc8ed279a289f9bc33929e8fa67c3129cd972bf1a490" exitCode=6
Jul 18 01:29:20.423061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:20.422648    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" event={"ID":"c4f5e571-1e96-4c6c-a737-75c91e925261","Type":"ContainerDied","Data":"d2586f5c3a5056e36081bc8ed279a289f9bc33929e8fa67c3129cd972bf1a490"}
Jul 18 01:29:20.423061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:20.422701    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" event={"ID":"c4f5e571-1e96-4c6c-a737-75c91e925261","Type":"ContainerStarted","Data":"08c61ea7aef9de363eba93eae3d2a8bb6bad2131fe1c46a9251739cf69857658"}
Jul 18 01:29:20.423061 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:20.422833    2554 scope.go:117] "RemoveContainer" containerID="d2586f5c3a5056e36081bc8ed279a289f9bc33929e8fa67c3129cd972bf1a490"
Jul 18 01:29:21.429385 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:21.429300    2554 generic.go:356] "Generic (PLEG): container finished" podID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerID="c691db16d786bd2f4c290b1429821c9793d8a63011fe2de7cb839d3dc9ef3bdc" exitCode=6
Jul 18 01:29:21.429876 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:21.429389    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" event={"ID":"c4f5e571-1e96-4c6c-a737-75c91e925261","Type":"ContainerDied","Data":"c691db16d786bd2f4c290b1429821c9793d8a63011fe2de7cb839d3dc9ef3bdc"}
Jul 18 01:29:21.429876 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:21.429455    2554 scope.go:117] "RemoveContainer" containerID="d2586f5c3a5056e36081bc8ed279a289f9bc33929e8fa67c3129cd972bf1a490"
Jul 18 01:29:21.429876 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:21.429658    2554 scope.go:117] "RemoveContainer" containerID="c691db16d786bd2f4c290b1429821c9793d8a63011fe2de7cb839d3dc9ef3bdc"
Jul 18 01:29:21.430040 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:29:21.429943    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd_odh-ai-gateway-infra(c4f5e571-1e96-4c6c-a737-75c91e925261)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261"
Jul 18 01:29:22.435802 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:22.435774    2554 scope.go:117] "RemoveContainer" containerID="c691db16d786bd2f4c290b1429821c9793d8a63011fe2de7cb839d3dc9ef3bdc"
Jul 18 01:29:22.436209 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:29:22.435964    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd_odh-ai-gateway-infra(c4f5e571-1e96-4c6c-a737-75c91e925261)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261"
Jul 18 01:29:23.563190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:23.563156    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"]
Jul 18 01:29:23.563649 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:23.563361    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy" containerID="cri-o://56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94" gracePeriod=30
Jul 18 01:29:24.423961 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.423920    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"]
Jul 18 01:29:24.424726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.424701    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:24.424917 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.424895    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="426d7dfd-4026-4bd4-853e-77394dc2eb84" containerName="revoke-keys"
Jul 18 01:29:24.428070 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.428054    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.430709 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.430691    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0192d7-openshift-default-dockercfg-4d4qm\""
Jul 18 01:29:24.439912 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.439891    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"]
Jul 18 01:29:24.448305 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448275    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-socket\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448407 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istiod-ca-cert\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448407 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448371    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-token\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448483 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448432    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-credential-socket\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448483 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448471    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-data\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448555 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448516    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-envoy\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448604 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448590    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crh8b\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-kube-api-access-crh8b\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448659 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448628    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-certs\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.448695 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.448671    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-podinfo\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549234 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-token\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549246    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-credential-socket\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549279    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-data\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549327    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-envoy\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549422 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crh8b\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-kube-api-access-crh8b\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549431    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-certs\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549461    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-podinfo\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549518    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-socket\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549717 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549549    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istiod-ca-cert\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549724    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-credential-socket\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549737    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-data\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.549905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-certs\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.550021 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.549992    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-socket\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.550290 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.550273    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istiod-ca-cert\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.551740 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.551720    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-envoy\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.551959 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.551943    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-podinfo\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.556683 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.556660    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-token\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.556928 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.556911    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crh8b\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-kube-api-access-crh8b\") pod \"e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") " pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.739662 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.739545    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:24.875902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.875876    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"]
Jul 18 01:29:24.878443 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:29:24.878407    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b8308e_1fb5_4306_bb7f_b06422cfe265.slice/crio-af1318d72188d658fb46b140920d9c7ea3110a58eaa6c2d234a38fad43178bfc WatchSource:0}: Error finding container af1318d72188d658fb46b140920d9c7ea3110a58eaa6c2d234a38fad43178bfc: Status 404 returned error can't find the container with id af1318d72188d658fb46b140920d9c7ea3110a58eaa6c2d234a38fad43178bfc
Jul 18 01:29:24.880808 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.880776    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:24.880888 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.880842    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:24.880888 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:24.880873    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:25.454610 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:25.454572    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" event={"ID":"e7b8308e-1fb5-4306-bb7f-b06422cfe265","Type":"ContainerStarted","Data":"a73c09582985031d285138da7578f9e93771a74796c3ccb94d81cdfc0d1ba663"}
Jul 18 01:29:25.454814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:25.454665    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" event={"ID":"e7b8308e-1fb5-4306-bb7f-b06422cfe265","Type":"ContainerStarted","Data":"af1318d72188d658fb46b140920d9c7ea3110a58eaa6c2d234a38fad43178bfc"}
Jul 18 01:29:25.474864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:25.474817    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" podStartSLOduration=1.474801317 podStartE2EDuration="1.474801317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:29:25.472694652 +0000 UTC m=+2921.598629632" watchObservedRunningTime="2026-07-18 01:29:25.474801317 +0000 UTC m=+2921.600736367"
Jul 18 01:29:25.740010 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:25.739920    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:25.745115 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:25.745090    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:26.460029 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:26.460001    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:26.461026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:26.461009    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:28.821601 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.821526    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:29:28.892815 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.892779    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-credential-socket\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.892815 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.892816    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-token\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893086 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.892869    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istiod-ca-cert\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893086 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.892889    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-certs\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893086 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.892917    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-podinfo\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893086 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.892996    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-data\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893086 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893073    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:28.893351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893142    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-socket\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893191    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-envoy\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893232    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxnvj\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-kube-api-access-lxnvj\") pod \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\" (UID: \"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a\") "
Jul 18 01:29:28.893351 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893280    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:28.893549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893347    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:28.893549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893464    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:28.893700 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893679    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.893753 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893705    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.893753 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893718    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.893860 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.893732    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.894524 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.894495    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:28.895438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.895411    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:29:28.895665 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.895640    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:29:28.895665 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.895654    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-kube-api-access-lxnvj" (OuterVolumeSpecName: "kube-api-access-lxnvj") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "kube-api-access-lxnvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:29:28.895837 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.895673    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" (UID: "3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:29:28.995122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.995073    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.995122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.995117    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.995122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.995126    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnvj\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-kube-api-access-lxnvj\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.995122 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.995136    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:28.995405 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:28.995145    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:29.477077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.477039    2554 generic.go:356] "Generic (PLEG): container finished" podID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerID="56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94" exitCode=0
Jul 18 01:29:29.477236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.477092    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"
Jul 18 01:29:29.477236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.477117    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" event={"ID":"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a","Type":"ContainerDied","Data":"56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94"}
Jul 18 01:29:29.477236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.477173    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf" event={"ID":"3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a","Type":"ContainerDied","Data":"3b0adbc202d7c9724cbc8127924f171fac3cf83e9ea6f3c9c40d467ad688d5aa"}
Jul 18 01:29:29.477236 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.477194    2554 scope.go:117] "RemoveContainer" containerID="56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94"
Jul 18 01:29:29.486506 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.486490    2554 scope.go:117] "RemoveContainer" containerID="97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee"
Jul 18 01:29:29.494683 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.494667    2554 scope.go:117] "RemoveContainer" containerID="56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94"
Jul 18 01:29:29.494923 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:29:29.494905    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94\": container with ID starting with 56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94 not found: ID does not exist" containerID="56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94"
Jul 18 01:29:29.494986 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.494929    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94"} err="failed to get container status \"56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94\": rpc error: code = NotFound desc = could not find container \"56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94\": container with ID starting with 56098ecbae8565b60898a0e651b1e9e9ca93e974872c7fd06cf09b18eedf2b94 not found: ID does not exist"
Jul 18 01:29:29.494986 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.494945    2554 scope.go:117] "RemoveContainer" containerID="97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee"
Jul 18 01:29:29.495185 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:29:29.495166    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee\": container with ID starting with 97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee not found: ID does not exist" containerID="97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee"
Jul 18 01:29:29.495229 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.495190    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee"} err="failed to get container status \"97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee\": rpc error: code = NotFound desc = could not find container \"97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee\": container with ID starting with 97bac54d3e6539ec559ebc6971ec50871bad29e4745affaf2e55746762e279ee not found: ID does not exist"
Jul 18 01:29:29.499639 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.499604    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"]
Jul 18 01:29:29.502954 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:29.502936    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a3406-openshift-default-54fbbdb44c-mgqwf"]
Jul 18 01:29:30.336573 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:30.336539    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" path="/var/lib/kubelet/pods/3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a/volumes"
Jul 18 01:29:32.586171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.586136    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"]
Jul 18 01:29:32.586582 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.586568    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy"
Jul 18 01:29:32.586666 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.586584    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy"
Jul 18 01:29:32.586713 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.586706    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy"
Jul 18 01:29:32.591358 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.591337    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.593878 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.593853    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-232dbd-openshift-default-dockercfg-sgwgm\""
Jul 18 01:29:32.603665 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.603640    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"]
Jul 18 01:29:32.726953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.726914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-credential-socket\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.726970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-envoy\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.727036    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17713509-33a4-47e1-9f55-10c1fc1f33a1-istiod-ca-cert\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.727093    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-certs\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727145 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.727113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-podinfo\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727301 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.727217    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-token\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727301 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.727255    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54x8g\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-kube-api-access-54x8g\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727301 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.727290    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-socket\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.727405 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.727323    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-data\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828082 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828045    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-token\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828090    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54x8g\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-kube-api-access-54x8g\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828132    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-socket\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828150    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-data\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828174    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-credential-socket\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828280 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828228    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-envoy\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828538 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17713509-33a4-47e1-9f55-10c1fc1f33a1-istiod-ca-cert\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828538 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828442    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-certs\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828538 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828468    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-podinfo\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828723 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828586    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-socket\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828723 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828637    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-data\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828719    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-credential-socket\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.828817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.828798    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-certs\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.829089 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.829067    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17713509-33a4-47e1-9f55-10c1fc1f33a1-istiod-ca-cert\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.830473 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.830455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-envoy\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.830905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.830886    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-podinfo\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.840566 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.840513    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-token\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.840674 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.840653    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54x8g\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-kube-api-access-54x8g\") pod \"e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") " pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:32.903019 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:32.902993    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:33.045059 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.044910    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"]
Jul 18 01:29:33.047598 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:29:33.047570    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17713509_33a4_47e1_9f55_10c1fc1f33a1.slice/crio-3788364de434ccf5feb582bc2d9b4908308c359257aedfb715f1dee76727fcc6 WatchSource:0}: Error finding container 3788364de434ccf5feb582bc2d9b4908308c359257aedfb715f1dee76727fcc6: Status 404 returned error can't find the container with id 3788364de434ccf5feb582bc2d9b4908308c359257aedfb715f1dee76727fcc6
Jul 18 01:29:33.050343 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.050303    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:33.050452 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.050371    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:33.050452 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.050401    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:33.502176 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.502134    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" event={"ID":"17713509-33a4-47e1-9f55-10c1fc1f33a1","Type":"ContainerStarted","Data":"02a8779f9b1e5480d7df6071deeba5bee19736d76c3847537cbf4819f2607101"}
Jul 18 01:29:33.502381 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.502204    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" event={"ID":"17713509-33a4-47e1-9f55-10c1fc1f33a1","Type":"ContainerStarted","Data":"3788364de434ccf5feb582bc2d9b4908308c359257aedfb715f1dee76727fcc6"}
Jul 18 01:29:33.557839 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.557788    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podStartSLOduration=1.557772146 podStartE2EDuration="1.557772146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:29:33.556554281 +0000 UTC m=+2929.682489263" watchObservedRunningTime="2026-07-18 01:29:33.557772146 +0000 UTC m=+2929.683707125"
Jul 18 01:29:33.903764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:33.903661    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:34.918184 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:34.918154    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:35.522046 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:35.522017    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:35.526392 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:35.526364    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:38.330893 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:38.330858    2554 scope.go:117] "RemoveContainer" containerID="c691db16d786bd2f4c290b1429821c9793d8a63011fe2de7cb839d3dc9ef3bdc"
Jul 18 01:29:39.545597 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:39.545562    2554 generic.go:356] "Generic (PLEG): container finished" podID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerID="9505162b50da26c73ed19193b1857b44cc17080f05cf0a8c7ab7e4661085120c" exitCode=6
Jul 18 01:29:39.546118 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:39.545638    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" event={"ID":"c4f5e571-1e96-4c6c-a737-75c91e925261","Type":"ContainerDied","Data":"9505162b50da26c73ed19193b1857b44cc17080f05cf0a8c7ab7e4661085120c"}
Jul 18 01:29:39.546118 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:39.545696    2554 scope.go:117] "RemoveContainer" containerID="c691db16d786bd2f4c290b1429821c9793d8a63011fe2de7cb839d3dc9ef3bdc"
Jul 18 01:29:39.546118 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:39.546017    2554 scope.go:117] "RemoveContainer" containerID="9505162b50da26c73ed19193b1857b44cc17080f05cf0a8c7ab7e4661085120c"
Jul 18 01:29:39.546325 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:29:39.546248    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd_odh-ai-gateway-infra(c4f5e571-1e96-4c6c-a737-75c91e925261)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261"
Jul 18 01:29:40.553176 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.553146    2554 generic.go:356] "Generic (PLEG): container finished" podID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerID="a73c09582985031d285138da7578f9e93771a74796c3ccb94d81cdfc0d1ba663" exitCode=0
Jul 18 01:29:40.553676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.553220    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" event={"ID":"e7b8308e-1fb5-4306-bb7f-b06422cfe265","Type":"ContainerDied","Data":"a73c09582985031d285138da7578f9e93771a74796c3ccb94d81cdfc0d1ba663"}
Jul 18 01:29:40.553768 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.553753    2554 scope.go:117] "RemoveContainer" containerID="a73c09582985031d285138da7578f9e93771a74796c3ccb94d81cdfc0d1ba663"
Jul 18 01:29:40.554848 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.554815    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:40.554959 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.554885    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:40.554959 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.554924    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:40.573799 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.573775    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"]
Jul 18 01:29:40.739950 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.739919    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"
Jul 18 01:29:40.812934 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.812844    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f5e571-1e96-4c6c-a737-75c91e925261-maas-api-service-ca\") pod \"c4f5e571-1e96-4c6c-a737-75c91e925261\" (UID: \"c4f5e571-1e96-4c6c-a737-75c91e925261\") "
Jul 18 01:29:40.813230 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.813208    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f5e571-1e96-4c6c-a737-75c91e925261-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4f5e571-1e96-4c6c-a737-75c91e925261" (UID: "c4f5e571-1e96-4c6c-a737-75c91e925261"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:40.914561 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:40.914518    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f5e571-1e96-4c6c-a737-75c91e925261-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:29:41.460096 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.460058    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:41.561694 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.561662    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"
Jul 18 01:29:41.562151 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.561692    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd" event={"ID":"c4f5e571-1e96-4c6c-a737-75c91e925261","Type":"ContainerDied","Data":"08c61ea7aef9de363eba93eae3d2a8bb6bad2131fe1c46a9251739cf69857658"}
Jul 18 01:29:41.562151 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.561760    2554 scope.go:117] "RemoveContainer" containerID="9505162b50da26c73ed19193b1857b44cc17080f05cf0a8c7ab7e4661085120c"
Jul 18 01:29:41.563856 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.563831    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" event={"ID":"e7b8308e-1fb5-4306-bb7f-b06422cfe265","Type":"ContainerStarted","Data":"021c38633eecddde8be467b3f10a7f95f7a300bf63440b1dcd2c8b229f548eb8"}
Jul 18 01:29:41.596376 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.596351    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"]
Jul 18 01:29:41.600486 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.600458    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9fpjd"]
Jul 18 01:29:41.740331 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.740303    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:41.741074 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.741052    2554 patch_prober.go:28] interesting pod/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 18 01:29:41.741201 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:41.741109    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 18 01:29:42.338184 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:42.338148    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" path="/var/lib/kubelet/pods/c4f5e571-1e96-4c6c-a737-75c91e925261/volumes"
Jul 18 01:29:42.740335 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:42.740298    2554 patch_prober.go:28] interesting pod/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 18 01:29:42.740764 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:42.740375    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 18 01:29:43.740406 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:43.740368    2554 patch_prober.go:28] interesting pod/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 18 01:29:43.740865 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:43.740433    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 18 01:29:44.740444 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:44.740392    2554 patch_prober.go:28] interesting pod/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 18 01:29:44.741024 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:44.740467    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 18 01:29:45.744285 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:45.744250    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:45.744739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:45.744536    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:45.745287 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:45.745267    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:29:51.476552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.475916    2554 patch_prober.go:28] interesting pod/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:48302->10.134.0.105:15021: read: connection reset by peer" start-of-body=
Jul 18 01:29:51.476552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.475981    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:48302->10.134.0.105:15021: read: connection reset by peer"
Jul 18 01:29:51.620568 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.620532    2554 generic.go:356] "Generic (PLEG): container finished" podID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerID="02a8779f9b1e5480d7df6071deeba5bee19736d76c3847537cbf4819f2607101" exitCode=0
Jul 18 01:29:51.620770 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.620608    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" event={"ID":"17713509-33a4-47e1-9f55-10c1fc1f33a1","Type":"ContainerDied","Data":"02a8779f9b1e5480d7df6071deeba5bee19736d76c3847537cbf4819f2607101"}
Jul 18 01:29:51.621118 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.621102    2554 scope.go:117] "RemoveContainer" containerID="02a8779f9b1e5480d7df6071deeba5bee19736d76c3847537cbf4819f2607101"
Jul 18 01:29:51.622194 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.622167    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:51.622278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.622234    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:51.622278 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:51.622265    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:29:52.627149 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:52.627115    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" event={"ID":"17713509-33a4-47e1-9f55-10c1fc1f33a1","Type":"ContainerStarted","Data":"d3a0d1c1ace283161c5feb2cca856dca9b9fbf955fbe4ee13bc9bd797b61eae6"}
Jul 18 01:29:52.903745 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:52.903643    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:52.903942 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:52.903915    2554 patch_prober.go:28] interesting pod/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Jul 18 01:29:52.904020 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:52.903975    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Jul 18 01:29:53.904099 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:53.904061    2554 patch_prober.go:28] interesting pod/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Jul 18 01:29:53.904499 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:53.904132    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Jul 18 01:29:54.904002 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:54.903969    2554 patch_prober.go:28] interesting pod/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Jul 18 01:29:54.904193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:54.904040    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Jul 18 01:29:55.903654 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:55.903598    2554 patch_prober.go:28] interesting pod/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Jul 18 01:29:55.903838 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:55.903676    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Jul 18 01:29:56.904269 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:56.904229    2554 patch_prober.go:28] interesting pod/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Jul 18 01:29:56.904791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:56.904306    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Jul 18 01:29:57.907797 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:57.907766    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:57.908248 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:57.908105    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:29:57.908840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:29:57.908820    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:30:13.585074 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:13.585039    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"]
Jul 18 01:30:13.585666 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:13.585299    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy" containerID="cri-o://021c38633eecddde8be467b3f10a7f95f7a300bf63440b1dcd2c8b229f548eb8" gracePeriod=30
Jul 18 01:30:15.747745 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:15.747706    2554 patch_prober.go:28] interesting pod/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 01:30:15.748226 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:15.747772    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 01:30:18.762301 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.762267    2554 generic.go:356] "Generic (PLEG): container finished" podID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerID="021c38633eecddde8be467b3f10a7f95f7a300bf63440b1dcd2c8b229f548eb8" exitCode=0
Jul 18 01:30:18.762734 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.762336    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" event={"ID":"e7b8308e-1fb5-4306-bb7f-b06422cfe265","Type":"ContainerDied","Data":"021c38633eecddde8be467b3f10a7f95f7a300bf63440b1dcd2c8b229f548eb8"}
Jul 18 01:30:18.762734 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.762398    2554 scope.go:117] "RemoveContainer" containerID="a73c09582985031d285138da7578f9e93771a74796c3ccb94d81cdfc0d1ba663"
Jul 18 01:30:18.836738 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.836706    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:30:18.907723 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.907604    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istiod-ca-cert\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.907723 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.907664    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-credential-socket\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.907886    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crh8b\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-kube-api-access-crh8b\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908192    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908255    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-socket\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908310    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-token\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908381    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-podinfo\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908423    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-data\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908423    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908438    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908470    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-envoy\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.908518    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-certs\") pod \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\" (UID: \"e7b8308e-1fb5-4306-bb7f-b06422cfe265\") "
Jul 18 01:30:18.909890 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.909842    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:18.910583 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.909917    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:18.911098 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911070    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-kube-api-access-crh8b" (OuterVolumeSpecName: "kube-api-access-crh8b") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "kube-api-access-crh8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:30:18.911206 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911148    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:30:18.911545 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911524    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:18.911647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911550    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:18.911647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911564    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:18.911647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911585    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:18.911647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911598    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crh8b\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-kube-api-access-crh8b\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:18.911647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911628    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:18.911647 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.911646    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:18.913679 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.912096    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:18.917371 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:18.917342    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7b8308e-1fb5-4306-bb7f-b06422cfe265" (UID: "e7b8308e-1fb5-4306-bb7f-b06422cfe265"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:30:19.012631 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.012590    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:19.012821 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.012646    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b8308e-1fb5-4306-bb7f-b06422cfe265-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:19.236500 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236421    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"]
Jul 18 01:30:19.236888 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236874    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy"
Jul 18 01:30:19.236930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236890    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy"
Jul 18 01:30:19.236930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236901    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy"
Jul 18 01:30:19.236930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236906    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy"
Jul 18 01:30:19.236930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236918    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:19.236930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236923    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:19.236930 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236929    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236934    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236945    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.236950    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.237005    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.237014    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" containerName="istio-proxy"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.237022    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.237028    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9c4ce5-94d1-4d43-bcd3-15e92cd7a90a" containerName="istio-proxy"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.237035    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:19.237119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.237043    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:19.240405 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.240387    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"
Jul 18 01:30:19.243266 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.243243    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:30:19.246044 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.246009    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"]
Jul 18 01:30:19.316192 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.316155    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e53f6-0f55-4c21-9848-c0bd20fe0d73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-2265r\" (UID: \"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"
Jul 18 01:30:19.417192 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.417156    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e53f6-0f55-4c21-9848-c0bd20fe0d73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-2265r\" (UID: \"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"
Jul 18 01:30:19.417832 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.417811    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e53f6-0f55-4c21-9848-c0bd20fe0d73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-2265r\" (UID: \"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"
Jul 18 01:30:19.553983 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.553950    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"
Jul 18 01:30:19.683440 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.683410    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"]
Jul 18 01:30:19.684994 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:30:19.684957    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3e53f6_0f55_4c21_9848_c0bd20fe0d73.slice/crio-2f020762933b450dcc2d983d7cd857f867ee2457b3ffea1d63aeb43dbfc50c87 WatchSource:0}: Error finding container 2f020762933b450dcc2d983d7cd857f867ee2457b3ffea1d63aeb43dbfc50c87: Status 404 returned error can't find the container with id 2f020762933b450dcc2d983d7cd857f867ee2457b3ffea1d63aeb43dbfc50c87
Jul 18 01:30:19.768590 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.768563    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"
Jul 18 01:30:19.769085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.768594    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5" event={"ID":"e7b8308e-1fb5-4306-bb7f-b06422cfe265","Type":"ContainerDied","Data":"af1318d72188d658fb46b140920d9c7ea3110a58eaa6c2d234a38fad43178bfc"}
Jul 18 01:30:19.769085 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.768674    2554 scope.go:117] "RemoveContainer" containerID="021c38633eecddde8be467b3f10a7f95f7a300bf63440b1dcd2c8b229f548eb8"
Jul 18 01:30:19.769971 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.769943    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" event={"ID":"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73","Type":"ContainerStarted","Data":"2f020762933b450dcc2d983d7cd857f867ee2457b3ffea1d63aeb43dbfc50c87"}
Jul 18 01:30:19.789968 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.789938    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"]
Jul 18 01:30:19.795180 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:19.795152    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0192d7-openshift-default-85f5dbf65c-rchk5"]
Jul 18 01:30:20.337724 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:20.337686    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b8308e-1fb5-4306-bb7f-b06422cfe265" path="/var/lib/kubelet/pods/e7b8308e-1fb5-4306-bb7f-b06422cfe265/volumes"
Jul 18 01:30:20.777946 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:20.777914    2554 generic.go:356] "Generic (PLEG): container finished" podID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerID="e27b96731ad285888f7543961d11c2cf9994de5573dfc07cf6aea07e832fe0d0" exitCode=6
Jul 18 01:30:20.778370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:20.777998    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" event={"ID":"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73","Type":"ContainerDied","Data":"e27b96731ad285888f7543961d11c2cf9994de5573dfc07cf6aea07e832fe0d0"}
Jul 18 01:30:20.778370 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:20.778182    2554 scope.go:117] "RemoveContainer" containerID="e27b96731ad285888f7543961d11c2cf9994de5573dfc07cf6aea07e832fe0d0"
Jul 18 01:30:21.785457 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:21.785423    2554 generic.go:356] "Generic (PLEG): container finished" podID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerID="3b9f5a98f2aa569944f8dace35224b8ec0eb8500f748601ce3227013a06c3ab7" exitCode=6
Jul 18 01:30:21.785871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:21.785507    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" event={"ID":"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73","Type":"ContainerDied","Data":"3b9f5a98f2aa569944f8dace35224b8ec0eb8500f748601ce3227013a06c3ab7"}
Jul 18 01:30:21.785871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:21.785562    2554 scope.go:117] "RemoveContainer" containerID="e27b96731ad285888f7543961d11c2cf9994de5573dfc07cf6aea07e832fe0d0"
Jul 18 01:30:21.785871 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:21.785710    2554 scope.go:117] "RemoveContainer" containerID="3b9f5a98f2aa569944f8dace35224b8ec0eb8500f748601ce3227013a06c3ab7"
Jul 18 01:30:21.786016 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:30:21.785992    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-2265r_odh-ai-gateway-infra(ba3e53f6-0f55-4c21-9848-c0bd20fe0d73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73"
Jul 18 01:30:22.792741 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:22.792715    2554 scope.go:117] "RemoveContainer" containerID="3b9f5a98f2aa569944f8dace35224b8ec0eb8500f748601ce3227013a06c3ab7"
Jul 18 01:30:22.793147 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:30:22.792929    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-2265r_odh-ai-gateway-infra(ba3e53f6-0f55-4c21-9848-c0bd20fe0d73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73"
Jul 18 01:30:37.330981 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:37.330942    2554 scope.go:117] "RemoveContainer" containerID="3b9f5a98f2aa569944f8dace35224b8ec0eb8500f748601ce3227013a06c3ab7"
Jul 18 01:30:37.872105 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:37.872020    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" event={"ID":"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73","Type":"ContainerStarted","Data":"59ddc6279b6da3ee8b998c40902a5126fa21e849521e78d60f0a172aa51042a2"}
Jul 18 01:30:37.888105 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:37.888053    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" podStartSLOduration=18.888038551 podStartE2EDuration="18.888038551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:37.886512296 +0000 UTC m=+2994.012447287" watchObservedRunningTime="2026-07-18 01:30:37.888038551 +0000 UTC m=+2994.013973528"
Jul 18 01:30:38.879102 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:38.879069    2554 generic.go:356] "Generic (PLEG): container finished" podID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerID="59ddc6279b6da3ee8b998c40902a5126fa21e849521e78d60f0a172aa51042a2" exitCode=6
Jul 18 01:30:38.879527 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:38.879152    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" event={"ID":"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73","Type":"ContainerDied","Data":"59ddc6279b6da3ee8b998c40902a5126fa21e849521e78d60f0a172aa51042a2"}
Jul 18 01:30:38.879527 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:38.879244    2554 scope.go:117] "RemoveContainer" containerID="3b9f5a98f2aa569944f8dace35224b8ec0eb8500f748601ce3227013a06c3ab7"
Jul 18 01:30:38.879527 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:38.879473    2554 scope.go:117] "RemoveContainer" containerID="59ddc6279b6da3ee8b998c40902a5126fa21e849521e78d60f0a172aa51042a2"
Jul 18 01:30:38.879751 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:30:38.879730    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-2265r_odh-ai-gateway-infra(ba3e53f6-0f55-4c21-9848-c0bd20fe0d73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73"
Jul 18 01:30:38.903028 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:38.902997    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"]
Jul 18 01:30:39.732565 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:39.732530    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"]
Jul 18 01:30:39.732812 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:39.732769    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" containerID="cri-o://d3a0d1c1ace283161c5feb2cca856dca9b9fbf955fbe4ee13bc9bd797b61eae6" gracePeriod=30
Jul 18 01:30:40.032180 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.032154    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"
Jul 18 01:30:40.119584 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.119547    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e53f6-0f55-4c21-9848-c0bd20fe0d73-maas-api-service-ca\") pod \"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73\" (UID: \"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73\") "
Jul 18 01:30:40.119996 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.119969    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3e53f6-0f55-4c21-9848-c0bd20fe0d73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" (UID: "ba3e53f6-0f55-4c21-9848-c0bd20fe0d73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:40.221174 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.221129    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e53f6-0f55-4c21-9848-c0bd20fe0d73-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:40.591328 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.591288    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"]
Jul 18 01:30:40.591908 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.591892    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:40.591953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.591911    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:40.591953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.591926    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:40.591953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.591934    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:40.592046 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.591974    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:40.592046 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.591983    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f5e571-1e96-4c6c-a737-75c91e925261" containerName="revoke-keys"
Jul 18 01:30:40.592108 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.592084    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:40.592108 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.592102    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:40.592171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.592110    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:40.595419 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.595367    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.598465 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.598444    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ad1b67-openshift-default-dockercfg-zfzwd\""
Jul 18 01:30:40.604953 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.604928    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"]
Jul 18 01:30:40.625281 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625246    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istiod-ca-cert\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625433 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625295    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-credential-socket\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625433 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-podinfo\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625433 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625377    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xxr\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-kube-api-access-d6xxr\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625433 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625402    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-token\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625434    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-socket\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625600 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-data\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625598    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-certs\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.625701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.625660    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-envoy\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726199 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726164    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-certs\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726199 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726202    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-envoy\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726241    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istiod-ca-cert\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726267    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-credential-socket\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726297    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-podinfo\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xxr\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-kube-api-access-d6xxr\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726358    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-token\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726439 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726394    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-socket\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726759 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-data\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726759 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726610    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-certs\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.726882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726859    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-credential-socket\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.727009 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.726990    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-data\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.727069 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.727013    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-socket\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.727069 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.727020    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istiod-ca-cert\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.728572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.728543    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-envoy\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.728877 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.728857    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-podinfo\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.733822 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.733801    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xxr\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-kube-api-access-d6xxr\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.733951 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.733836    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-token\") pod \"e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") " pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.893713 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.893604    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"
Jul 18 01:30:40.893713 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.893604    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r" event={"ID":"ba3e53f6-0f55-4c21-9848-c0bd20fe0d73","Type":"ContainerDied","Data":"2f020762933b450dcc2d983d7cd857f867ee2457b3ffea1d63aeb43dbfc50c87"}
Jul 18 01:30:40.893713 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.893698    2554 scope.go:117] "RemoveContainer" containerID="59ddc6279b6da3ee8b998c40902a5126fa21e849521e78d60f0a172aa51042a2"
Jul 18 01:30:40.907452 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.907420    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:40.909521 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.909498    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"]
Jul 18 01:30:40.915447 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:40.915422    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-2265r"]
Jul 18 01:30:41.044458 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.044432    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"]
Jul 18 01:30:41.045522 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:30:41.045485    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33ed5e6_bd44_49d5_bbb5_8ba28f1dd971.slice/crio-b922271d62d995225cb46c89c49708242abd2aa17efa2ccdf849d0603b9937a8 WatchSource:0}: Error finding container b922271d62d995225cb46c89c49708242abd2aa17efa2ccdf849d0603b9937a8: Status 404 returned error can't find the container with id b922271d62d995225cb46c89c49708242abd2aa17efa2ccdf849d0603b9937a8
Jul 18 01:30:41.047749 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.047720    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:41.047842 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.047796    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:41.047911 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.047842    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:41.901840 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.901807    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" event={"ID":"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971","Type":"ContainerStarted","Data":"d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322"}
Jul 18 01:30:41.902037 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.901857    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" event={"ID":"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971","Type":"ContainerStarted","Data":"b922271d62d995225cb46c89c49708242abd2aa17efa2ccdf849d0603b9937a8"}
Jul 18 01:30:41.907461 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.907445    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:41.925582 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:41.925536    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" podStartSLOduration=1.9255214889999999 podStartE2EDuration="1.925521489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:41.923317005 +0000 UTC m=+2998.049251989" watchObservedRunningTime="2026-07-18 01:30:41.925521489 +0000 UTC m=+2998.051456468"
Jul 18 01:30:42.336339 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:42.336303    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" path="/var/lib/kubelet/pods/ba3e53f6-0f55-4c21-9848-c0bd20fe0d73/volumes"
Jul 18 01:30:42.910442 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:42.910408    2554 patch_prober.go:28] interesting pod/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 01:30:42.910656 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:42.910466    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 01:30:42.912709 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:42.912685    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:43.912905 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:43.912864    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:43.913992 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:43.913967    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:44.921031 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:44.920994    2554 generic.go:356] "Generic (PLEG): container finished" podID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerID="d3a0d1c1ace283161c5feb2cca856dca9b9fbf955fbe4ee13bc9bd797b61eae6" exitCode=0
Jul 18 01:30:44.921477 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:44.921066    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" event={"ID":"17713509-33a4-47e1-9f55-10c1fc1f33a1","Type":"ContainerDied","Data":"d3a0d1c1ace283161c5feb2cca856dca9b9fbf955fbe4ee13bc9bd797b61eae6"}
Jul 18 01:30:44.921477 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:44.921146    2554 scope.go:117] "RemoveContainer" containerID="02a8779f9b1e5480d7df6071deeba5bee19736d76c3847537cbf4819f2607101"
Jul 18 01:30:44.988672 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:44.988642    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:30:45.071862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.071764    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54x8g\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-kube-api-access-54x8g\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.071862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.071826    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-token\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.071862 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.071853    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-socket\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.072155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.071987    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-podinfo\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.072155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072069    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17713509-33a4-47e1-9f55-10c1fc1f33a1-istiod-ca-cert\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.072155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072112    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-credential-socket\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.072155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072138    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-data\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.072155 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072147    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:45.072417 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072221    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-envoy\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.072417 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072259    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-certs\") pod \"17713509-33a4-47e1-9f55-10c1fc1f33a1\" (UID: \"17713509-33a4-47e1-9f55-10c1fc1f33a1\") "
Jul 18 01:30:45.072417 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072360    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:45.072714 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072689    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.072831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072718    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.072831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072694    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:45.072951 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.072920    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17713509-33a4-47e1-9f55-10c1fc1f33a1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:45.073161 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.073133    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-data" (OuterVolumeSpecName: "istio-data") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:45.074279 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.074247    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-token" (OuterVolumeSpecName: "istio-token") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:30:45.074393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.074336    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:30:45.074524 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.074502    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:30:45.074524 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.074508    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-kube-api-access-54x8g" (OuterVolumeSpecName: "kube-api-access-54x8g") pod "17713509-33a4-47e1-9f55-10c1fc1f33a1" (UID: "17713509-33a4-47e1-9f55-10c1fc1f33a1"). InnerVolumeSpecName "kube-api-access-54x8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:30:45.173311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.173275    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.173311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.173306    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.173311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.173315    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17713509-33a4-47e1-9f55-10c1fc1f33a1-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.173311 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.173323    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.173643 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.173332    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.173643 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.173339    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17713509-33a4-47e1-9f55-10c1fc1f33a1-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.173643 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.173347    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54x8g\" (UniqueName: \"kubernetes.io/projected/17713509-33a4-47e1-9f55-10c1fc1f33a1-kube-api-access-54x8g\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:30:45.926695 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.926655    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm" event={"ID":"17713509-33a4-47e1-9f55-10c1fc1f33a1","Type":"ContainerDied","Data":"3788364de434ccf5feb582bc2d9b4908308c359257aedfb715f1dee76727fcc6"}
Jul 18 01:30:45.926695 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.926686    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"
Jul 18 01:30:45.927190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.926726    2554 scope.go:117] "RemoveContainer" containerID="d3a0d1c1ace283161c5feb2cca856dca9b9fbf955fbe4ee13bc9bd797b61eae6"
Jul 18 01:30:45.948093 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.948064    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"]
Jul 18 01:30:45.952942 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:45.952916    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-232dbd-openshift-default-d6ccfc6b7-cgqcm"]
Jul 18 01:30:46.336777 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:46.336739    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" path="/var/lib/kubelet/pods/17713509-33a4-47e1-9f55-10c1fc1f33a1/volumes"
Jul 18 01:30:47.315453 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315414    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"]
Jul 18 01:30:47.315947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315874    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy"
Jul 18 01:30:47.315947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315887    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy"
Jul 18 01:30:47.315947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315897    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy"
Jul 18 01:30:47.315947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315904    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy"
Jul 18 01:30:47.315947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315916    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:47.315947 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315921    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3e53f6-0f55-4c21-9848-c0bd20fe0d73" containerName="revoke-keys"
Jul 18 01:30:47.316142 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.315990    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy"
Jul 18 01:30:47.320183 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.320165    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"
Jul 18 01:30:47.323330 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.323307    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:30:47.325377 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.325349    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"]
Jul 18 01:30:47.396335 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.396294    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5200da-0137-41d5-8ba2-4c86e283dea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g\" (UID: \"5f5200da-0137-41d5-8ba2-4c86e283dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"
Jul 18 01:30:47.497291 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.497239    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5200da-0137-41d5-8ba2-4c86e283dea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g\" (UID: \"5f5200da-0137-41d5-8ba2-4c86e283dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"
Jul 18 01:30:47.497904 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.497881    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5200da-0137-41d5-8ba2-4c86e283dea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g\" (UID: \"5f5200da-0137-41d5-8ba2-4c86e283dea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"
Jul 18 01:30:47.634395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.634296    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"
Jul 18 01:30:47.766008 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.765972    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"]
Jul 18 01:30:47.766366 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:30:47.766347    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5200da_0137_41d5_8ba2_4c86e283dea5.slice/crio-4602f60aee1a34e913ec2565f8a66b1de333ae9fa57ff454e8ca3e0c513f4912 WatchSource:0}: Error finding container 4602f60aee1a34e913ec2565f8a66b1de333ae9fa57ff454e8ca3e0c513f4912: Status 404 returned error can't find the container with id 4602f60aee1a34e913ec2565f8a66b1de333ae9fa57ff454e8ca3e0c513f4912
Jul 18 01:30:47.941738 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:47.941654    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" event={"ID":"5f5200da-0137-41d5-8ba2-4c86e283dea5","Type":"ContainerStarted","Data":"4602f60aee1a34e913ec2565f8a66b1de333ae9fa57ff454e8ca3e0c513f4912"}
Jul 18 01:30:48.949262 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.949222    2554 generic.go:356] "Generic (PLEG): container finished" podID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerID="7e276bff882aa8405757c49b26b37b1ec4a7644adeb757faee523822a31f85d6" exitCode=6
Jul 18 01:30:48.949719 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.949312    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" event={"ID":"5f5200da-0137-41d5-8ba2-4c86e283dea5","Type":"ContainerDied","Data":"7e276bff882aa8405757c49b26b37b1ec4a7644adeb757faee523822a31f85d6"}
Jul 18 01:30:48.949719 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.949493    2554 scope.go:117] "RemoveContainer" containerID="7e276bff882aa8405757c49b26b37b1ec4a7644adeb757faee523822a31f85d6"
Jul 18 01:30:48.977353 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.977320    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"]
Jul 18 01:30:48.977849 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.977834    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="17713509-33a4-47e1-9f55-10c1fc1f33a1" containerName="istio-proxy"
Jul 18 01:30:48.982100 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.982078    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:48.984763 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.984741    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-0f7fdc-openshift-default-dockercfg-tc224\""
Jul 18 01:30:48.991761 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:48.991737    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"]
Jul 18 01:30:49.112473 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112440    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-credential-socket\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112609 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112498    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-socket\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112609 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112542    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-certs\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112711 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112640    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-istio-token\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112711 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112703    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfn7c\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-kube-api-access-jfn7c\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112730    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7178e0-2905-4913-b07f-73b453f0e993-istio-podinfo\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112791 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112757    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7178e0-2905-4913-b07f-73b453f0e993-istiod-ca-cert\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112863 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112828    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-data\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.112915 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.112894    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-envoy\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214011    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-socket\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214107 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214078    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-certs\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-istio-token\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214186    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfn7c\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-kube-api-access-jfn7c\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214217    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7178e0-2905-4913-b07f-73b453f0e993-istio-podinfo\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214517 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7178e0-2905-4913-b07f-73b453f0e993-istiod-ca-cert\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214569 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214533    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-certs\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.214641 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214559    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-socket\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.215024 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.214999    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-data\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.215150 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.215061    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-envoy\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.215150 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.215096    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-credential-socket\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.215362 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.215336    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-data\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.215475 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.215401    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-credential-socket\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.215626 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.215588    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7178e0-2905-4913-b07f-73b453f0e993-istiod-ca-cert\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.217471 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.217447    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7178e0-2905-4913-b07f-73b453f0e993-istio-podinfo\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.217591 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.217547    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-envoy\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.222427 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.222394    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-istio-token\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.222553 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.222535    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfn7c\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-kube-api-access-jfn7c\") pod \"e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") " pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.299813 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.299766    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:49.444244 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.444220    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"]
Jul 18 01:30:49.446122 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:30:49.446100    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7178e0_2905_4913_b07f_73b453f0e993.slice/crio-74ae506dfa6c5c9046aa7089d13096666503a196e4f0bdc21517ba1a47ee8909 WatchSource:0}: Error finding container 74ae506dfa6c5c9046aa7089d13096666503a196e4f0bdc21517ba1a47ee8909: Status 404 returned error can't find the container with id 74ae506dfa6c5c9046aa7089d13096666503a196e4f0bdc21517ba1a47ee8909
Jul 18 01:30:49.448329 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.448299    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:49.448399 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.448361    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:49.448399 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.448390    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:49.956763 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.956727    2554 generic.go:356] "Generic (PLEG): container finished" podID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerID="cef6485119a7ad54ec97ae3d223b73c6ab15742174c957494e748885486cc678" exitCode=6
Jul 18 01:30:49.957240 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.956811    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" event={"ID":"5f5200da-0137-41d5-8ba2-4c86e283dea5","Type":"ContainerDied","Data":"cef6485119a7ad54ec97ae3d223b73c6ab15742174c957494e748885486cc678"}
Jul 18 01:30:49.957240 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.956887    2554 scope.go:117] "RemoveContainer" containerID="7e276bff882aa8405757c49b26b37b1ec4a7644adeb757faee523822a31f85d6"
Jul 18 01:30:49.957240 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.957164    2554 scope.go:117] "RemoveContainer" containerID="cef6485119a7ad54ec97ae3d223b73c6ab15742174c957494e748885486cc678"
Jul 18 01:30:49.957466 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:30:49.957442    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g_odh-ai-gateway-infra(5f5200da-0137-41d5-8ba2-4c86e283dea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5"
Jul 18 01:30:49.958691 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.958604    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" event={"ID":"cc7178e0-2905-4913-b07f-73b453f0e993","Type":"ContainerStarted","Data":"57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00"}
Jul 18 01:30:49.958782 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.958727    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" event={"ID":"cc7178e0-2905-4913-b07f-73b453f0e993","Type":"ContainerStarted","Data":"74ae506dfa6c5c9046aa7089d13096666503a196e4f0bdc21517ba1a47ee8909"}
Jul 18 01:30:49.992193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:49.992133    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" podStartSLOduration=1.992115552 podStartE2EDuration="1.992115552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:49.990881805 +0000 UTC m=+3006.116816820" watchObservedRunningTime="2026-07-18 01:30:49.992115552 +0000 UTC m=+3006.118050583"
Jul 18 01:30:50.300963 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:50.300926    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:50.309931 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:50.309892    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:50.965185 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:50.965152    2554 scope.go:117] "RemoveContainer" containerID="cef6485119a7ad54ec97ae3d223b73c6ab15742174c957494e748885486cc678"
Jul 18 01:30:50.965667 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:50.965235    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:50.965667 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:30:50.965382    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g_odh-ai-gateway-infra(5f5200da-0137-41d5-8ba2-4c86e283dea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5"
Jul 18 01:30:50.966292 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:50.966272    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:30:58.003068 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:58.003029    2554 generic.go:356] "Generic (PLEG): container finished" podID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerID="d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322" exitCode=0
Jul 18 01:30:58.003523 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:58.003110    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" event={"ID":"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971","Type":"ContainerDied","Data":"d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322"}
Jul 18 01:30:58.003571 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:58.003553    2554 scope.go:117] "RemoveContainer" containerID="d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322"
Jul 18 01:30:58.005053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:58.004729    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:58.005053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:58.004820    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:58.005053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:58.004870    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:30:58.913593 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:58.913553    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:59.015412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:59.015351    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" event={"ID":"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971","Type":"ContainerStarted","Data":"21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0"}
Jul 18 01:30:59.909331 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:59.909292    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:30:59.909552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:59.909422    2554 patch_prober.go:28] interesting pod/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 18 01:30:59.909552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:30:59.909467    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 18 01:31:00.908326 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:00.908286    2554 patch_prober.go:28] interesting pod/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 18 01:31:00.908981 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:00.908947    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 18 01:31:01.907973 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:01.907922    2554 patch_prober.go:28] interesting pod/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 18 01:31:01.908203 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:01.908000    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 18 01:31:02.334882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:02.334849    2554 scope.go:117] "RemoveContainer" containerID="cef6485119a7ad54ec97ae3d223b73c6ab15742174c957494e748885486cc678"
Jul 18 01:31:02.908885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:02.908782    2554 patch_prober.go:28] interesting pod/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 18 01:31:02.908885 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:02.908857    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 18 01:31:03.052300 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:03.052263    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" event={"ID":"5f5200da-0137-41d5-8ba2-4c86e283dea5","Type":"ContainerStarted","Data":"aa7a5db51932fdded1d0631af783730bdeaebd87e7fbd0a5342d6217d17a0bf0"}
Jul 18 01:31:03.070193 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:03.070141    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" podStartSLOduration=16.070125003 podStartE2EDuration="16.070125003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:31:03.068238546 +0000 UTC m=+3019.194173527" watchObservedRunningTime="2026-07-18 01:31:03.070125003 +0000 UTC m=+3019.196059981"
Jul 18 01:31:03.912121 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:03.912082    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:31:03.912582 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:03.912558    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:31:03.913249 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:03.913228    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:31:04.060526 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:04.060498    2554 generic.go:356] "Generic (PLEG): container finished" podID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerID="aa7a5db51932fdded1d0631af783730bdeaebd87e7fbd0a5342d6217d17a0bf0" exitCode=6
Jul 18 01:31:04.060745 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:04.060574    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" event={"ID":"5f5200da-0137-41d5-8ba2-4c86e283dea5","Type":"ContainerDied","Data":"aa7a5db51932fdded1d0631af783730bdeaebd87e7fbd0a5342d6217d17a0bf0"}
Jul 18 01:31:04.060745 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:04.060665    2554 scope.go:117] "RemoveContainer" containerID="cef6485119a7ad54ec97ae3d223b73c6ab15742174c957494e748885486cc678"
Jul 18 01:31:04.061021 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:04.061000    2554 scope.go:117] "RemoveContainer" containerID="aa7a5db51932fdded1d0631af783730bdeaebd87e7fbd0a5342d6217d17a0bf0"
Jul 18 01:31:04.061249 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:31:04.061221    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g_odh-ai-gateway-infra(5f5200da-0137-41d5-8ba2-4c86e283dea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5"
Jul 18 01:31:04.078157 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:04.078133    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"]
Jul 18 01:31:05.069119 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.069090    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc7178e0-2905-4913-b07f-73b453f0e993" containerID="57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00" exitCode=0
Jul 18 01:31:05.069469 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.069166    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" event={"ID":"cc7178e0-2905-4913-b07f-73b453f0e993","Type":"ContainerDied","Data":"57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00"}
Jul 18 01:31:05.069713 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.069697    2554 scope.go:117] "RemoveContainer" containerID="57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00"
Jul 18 01:31:05.070668 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.070603    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:31:05.070827 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.070708    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:31:05.070827 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.070747    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 18 01:31:05.236185 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.236157    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"
Jul 18 01:31:05.308147 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.308056    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5200da-0137-41d5-8ba2-4c86e283dea5-maas-api-service-ca\") pod \"5f5200da-0137-41d5-8ba2-4c86e283dea5\" (UID: \"5f5200da-0137-41d5-8ba2-4c86e283dea5\") "
Jul 18 01:31:05.308412 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.308389    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5200da-0137-41d5-8ba2-4c86e283dea5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f5200da-0137-41d5-8ba2-4c86e283dea5" (UID: "5f5200da-0137-41d5-8ba2-4c86e283dea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:31:05.409082 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.409043    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5200da-0137-41d5-8ba2-4c86e283dea5-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:31:05.965254 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:05.965210    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:31:06.082841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.082809    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"
Jul 18 01:31:06.082841 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.082816    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g" event={"ID":"5f5200da-0137-41d5-8ba2-4c86e283dea5","Type":"ContainerDied","Data":"4602f60aee1a34e913ec2565f8a66b1de333ae9fa57ff454e8ca3e0c513f4912"}
Jul 18 01:31:06.083364 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.082914    2554 scope.go:117] "RemoveContainer" containerID="aa7a5db51932fdded1d0631af783730bdeaebd87e7fbd0a5342d6217d17a0bf0"
Jul 18 01:31:06.085315 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.085290    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" event={"ID":"cc7178e0-2905-4913-b07f-73b453f0e993","Type":"ContainerStarted","Data":"92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d"}
Jul 18 01:31:06.137371 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.137329    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"]
Jul 18 01:31:06.140817 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.140787    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-mkw6g"]
Jul 18 01:31:06.300820 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.300782    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:31:06.301926 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.301902    2554 patch_prober.go:28] interesting pod/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Jul 18 01:31:06.302045 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.301958    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Jul 18 01:31:06.340864 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:06.340821    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" path="/var/lib/kubelet/pods/5f5200da-0137-41d5-8ba2-4c86e283dea5/volumes"
Jul 18 01:31:07.300962 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:07.300925    2554 patch_prober.go:28] interesting pod/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Jul 18 01:31:07.301389 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:07.300986    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Jul 18 01:31:08.300513 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:08.300475    2554 patch_prober.go:28] interesting pod/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Jul 18 01:31:08.300739 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:08.300534    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Jul 18 01:31:09.300742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:09.300706    2554 patch_prober.go:28] interesting pod/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Jul 18 01:31:09.301148 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:09.300775    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Jul 18 01:31:10.304258 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:10.304224    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:31:10.304672 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:10.304510    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:31:10.305087 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:31:10.305072    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:37:30.716866 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:30.716830    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"]
Jul 18 01:37:30.717395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:30.717076    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy" containerID="cri-o://21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0" gracePeriod=30
Jul 18 01:37:33.915151 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:33.915109    2554 patch_prober.go:28] interesting pod/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 01:37:33.915536 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:33.915184    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 01:37:35.738348 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:37:35.738308    2554 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33ed5e6_bd44_49d5_bbb5_8ba28f1dd971.slice/crio-conmon-21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0.scope\": RecentStats: unable to find data in memory cache]"
Jul 18 01:37:35.970603 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:35.970577    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:37:36.061170 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.061131    2554 generic.go:356] "Generic (PLEG): container finished" podID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerID="21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0" exitCode=0
Jul 18 01:37:36.061345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.061180    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"
Jul 18 01:37:36.061345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.061215    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" event={"ID":"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971","Type":"ContainerDied","Data":"21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0"}
Jul 18 01:37:36.061345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.061286    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2" event={"ID":"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971","Type":"ContainerDied","Data":"b922271d62d995225cb46c89c49708242abd2aa17efa2ccdf849d0603b9937a8"}
Jul 18 01:37:36.061345 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.061308    2554 scope.go:117] "RemoveContainer" containerID="21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0"
Jul 18 01:37:36.064566 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064550    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-data\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.064703 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064577    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-socket\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.064703 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064602    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-credential-socket\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.064703 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064648    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-token\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.064882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064688    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-certs\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.064882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064764    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6xxr\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-kube-api-access-d6xxr\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.064882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064808    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-envoy\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.064882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064855    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:37:36.064882 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064863    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istiod-ca-cert\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.065153 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064921    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:37:36.065153 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.064955    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-podinfo\") pod \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\" (UID: \"b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971\") "
Jul 18 01:37:36.065528 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.065347    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.065528 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.065372    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.065528 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.065423    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:37:36.065742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.065526    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:36.065988 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.065948    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-data" (OuterVolumeSpecName: "istio-data") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:37:36.067402 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.067373    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-token" (OuterVolumeSpecName: "istio-token") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:37:36.067402 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.067390    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:37:36.067546 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.067423    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-kube-api-access-d6xxr" (OuterVolumeSpecName: "kube-api-access-d6xxr") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "kube-api-access-d6xxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:37:36.067546 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.067519    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" (UID: "b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:37:36.071148 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.071131    2554 scope.go:117] "RemoveContainer" containerID="d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322"
Jul 18 01:37:36.086041 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.086020    2554 scope.go:117] "RemoveContainer" containerID="21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0"
Jul 18 01:37:36.086327 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:37:36.086306    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0\": container with ID starting with 21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0 not found: ID does not exist" containerID="21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0"
Jul 18 01:37:36.086395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.086335    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0"} err="failed to get container status \"21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0\": rpc error: code = NotFound desc = could not find container \"21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0\": container with ID starting with 21b2d1e429f8ca0a690e9eb726f5b5c253d94dba0b1b5631d51501ffb0b8caa0 not found: ID does not exist"
Jul 18 01:37:36.086395 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.086355    2554 scope.go:117] "RemoveContainer" containerID="d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322"
Jul 18 01:37:36.086596 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:37:36.086579    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322\": container with ID starting with d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322 not found: ID does not exist" containerID="d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322"
Jul 18 01:37:36.086656 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.086600    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322"} err="failed to get container status \"d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322\": rpc error: code = NotFound desc = could not find container \"d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322\": container with ID starting with d6eca35d6d7a56101254e69961cad18f598350ff9d556354528aa2656df43322 not found: ID does not exist"
Jul 18 01:37:36.166190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.166163    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.166190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.166185    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.166190 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.166194    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.166404 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.166204    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6xxr\" (UniqueName: \"kubernetes.io/projected/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-kube-api-access-d6xxr\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.166404 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.166214    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.166404 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.166223    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.166404 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.166230    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:37:36.380274 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.380243    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"]
Jul 18 01:37:36.383861 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:36.383836    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ad1b67-openshift-default-7bf784f475-7t9r2"]
Jul 18 01:37:38.336354 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:38.336319    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" path="/var/lib/kubelet/pods/b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971/volumes"
Jul 18 01:37:55.629221 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629186    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"]
Jul 18 01:37:55.629664 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629610    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy"
Jul 18 01:37:55.629664 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629639    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy"
Jul 18 01:37:55.629664 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629651    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy"
Jul 18 01:37:55.629664 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629657    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629668    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629674    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629683    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629689    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629766    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629776    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629786    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:55.629814 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.629796    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33ed5e6-bd44-49d5-bbb5-8ba28f1dd971" containerName="istio-proxy"
Jul 18 01:37:55.634752 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.634730    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"
Jul 18 01:37:55.637313 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.637289    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:37:55.639676 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.639606    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"]
Jul 18 01:37:55.657197 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.657168    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b490bc-772b-4d77-bfe8-f8766ec236ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-56qnl\" (UID: \"30b490bc-772b-4d77-bfe8-f8766ec236ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"
Jul 18 01:37:55.758231 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.758191    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b490bc-772b-4d77-bfe8-f8766ec236ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-56qnl\" (UID: \"30b490bc-772b-4d77-bfe8-f8766ec236ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"
Jul 18 01:37:55.758850 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.758831    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b490bc-772b-4d77-bfe8-f8766ec236ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-56qnl\" (UID: \"30b490bc-772b-4d77-bfe8-f8766ec236ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"
Jul 18 01:37:55.950048 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:55.949968    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"
Jul 18 01:37:56.083920 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:56.083897    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"]
Jul 18 01:37:56.086149 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:37:56.086119    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b490bc_772b_4d77_bfe8_f8766ec236ad.slice/crio-a6c4a3a6337d87e6f8256fc3595ddb496085664d507d0da61959899c4ff68b8e WatchSource:0}: Error finding container a6c4a3a6337d87e6f8256fc3595ddb496085664d507d0da61959899c4ff68b8e: Status 404 returned error can't find the container with id a6c4a3a6337d87e6f8256fc3595ddb496085664d507d0da61959899c4ff68b8e
Jul 18 01:37:56.164141 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:56.164108    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" event={"ID":"30b490bc-772b-4d77-bfe8-f8766ec236ad","Type":"ContainerStarted","Data":"a6c4a3a6337d87e6f8256fc3595ddb496085664d507d0da61959899c4ff68b8e"}
Jul 18 01:37:56.788374 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:56.788337    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"]
Jul 18 01:37:56.788780 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:56.788640    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy" containerID="cri-o://92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d" gracePeriod=30
Jul 18 01:37:57.171054 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:57.170960    2554 generic.go:356] "Generic (PLEG): container finished" podID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerID="99cc179c0bf4a0af140b176eca4a5750da6d43ab8f13d1b8d78ec3c461b1d4e4" exitCode=6
Jul 18 01:37:57.171054 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:57.170999    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" event={"ID":"30b490bc-772b-4d77-bfe8-f8766ec236ad","Type":"ContainerDied","Data":"99cc179c0bf4a0af140b176eca4a5750da6d43ab8f13d1b8d78ec3c461b1d4e4"}
Jul 18 01:37:57.171277 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:57.171200    2554 scope.go:117] "RemoveContainer" containerID="99cc179c0bf4a0af140b176eca4a5750da6d43ab8f13d1b8d78ec3c461b1d4e4"
Jul 18 01:37:58.180187 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:58.180153    2554 generic.go:356] "Generic (PLEG): container finished" podID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerID="fb8a02a14c6eca35922421d05ae7128ebc7366d21d35f70eb2073a9662ecc008" exitCode=6
Jul 18 01:37:58.180606 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:58.180215    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" event={"ID":"30b490bc-772b-4d77-bfe8-f8766ec236ad","Type":"ContainerDied","Data":"fb8a02a14c6eca35922421d05ae7128ebc7366d21d35f70eb2073a9662ecc008"}
Jul 18 01:37:58.180606 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:58.180263    2554 scope.go:117] "RemoveContainer" containerID="99cc179c0bf4a0af140b176eca4a5750da6d43ab8f13d1b8d78ec3c461b1d4e4"
Jul 18 01:37:58.180606 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:58.180475    2554 scope.go:117] "RemoveContainer" containerID="fb8a02a14c6eca35922421d05ae7128ebc7366d21d35f70eb2073a9662ecc008"
Jul 18 01:37:58.180778 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:37:58.180759    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-56qnl_odh-ai-gateway-infra(30b490bc-772b-4d77-bfe8-f8766ec236ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad"
Jul 18 01:37:59.186849 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.186819    2554 scope.go:117] "RemoveContainer" containerID="fb8a02a14c6eca35922421d05ae7128ebc7366d21d35f70eb2073a9662ecc008"
Jul 18 01:37:59.187234 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:37:59.187009    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-56qnl_odh-ai-gateway-infra(30b490bc-772b-4d77-bfe8-f8766ec236ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad"
Jul 18 01:37:59.577952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.577916    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"]
Jul 18 01:37:59.578437 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.578420    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:59.578492 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.578441    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:59.578526 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.578511    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5200da-0137-41d5-8ba2-4c86e283dea5" containerName="revoke-keys"
Jul 18 01:37:59.582686 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.582671    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"
Jul 18 01:37:59.587897 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.587874    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"]
Jul 18 01:37:59.594274 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.594244    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f134418f-3d29-496b-8abd-897714961717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p\" (UID: \"f134418f-3d29-496b-8abd-897714961717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"
Jul 18 01:37:59.695524 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.695490    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f134418f-3d29-496b-8abd-897714961717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p\" (UID: \"f134418f-3d29-496b-8abd-897714961717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"
Jul 18 01:37:59.696141 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.696124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f134418f-3d29-496b-8abd-897714961717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p\" (UID: \"f134418f-3d29-496b-8abd-897714961717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"
Jul 18 01:37:59.894830 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:37:59.894748    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"
Jul 18 01:38:00.023678 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:00.023649    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"]
Jul 18 01:38:00.026056 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:38:00.026025    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf134418f_3d29_496b_8abd_897714961717.slice/crio-46579724b8829616584c103cd7a246f9d00242b51fd9747ebfa0aecd93826751 WatchSource:0}: Error finding container 46579724b8829616584c103cd7a246f9d00242b51fd9747ebfa0aecd93826751: Status 404 returned error can't find the container with id 46579724b8829616584c103cd7a246f9d00242b51fd9747ebfa0aecd93826751
Jul 18 01:38:00.192972 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:00.192886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" event={"ID":"f134418f-3d29-496b-8abd-897714961717","Type":"ContainerStarted","Data":"46579724b8829616584c103cd7a246f9d00242b51fd9747ebfa0aecd93826751"}
Jul 18 01:38:01.199443 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:01.199408    2554 generic.go:356] "Generic (PLEG): container finished" podID="f134418f-3d29-496b-8abd-897714961717" containerID="26118cd0e72111a543322a8a7a8a67837941063c476df4be327df282261247e9" exitCode=6
Jul 18 01:38:01.199911 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:01.199493    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" event={"ID":"f134418f-3d29-496b-8abd-897714961717","Type":"ContainerDied","Data":"26118cd0e72111a543322a8a7a8a67837941063c476df4be327df282261247e9"}
Jul 18 01:38:01.199911 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:01.199751    2554 scope.go:117] "RemoveContainer" containerID="26118cd0e72111a543322a8a7a8a67837941063c476df4be327df282261247e9"
Jul 18 01:38:02.056533 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.056466    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:38:02.120726 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.120697    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-data\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.120887 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.120739    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7178e0-2905-4913-b07f-73b453f0e993-istiod-ca-cert\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.120887 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.120799    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-credential-socket\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.120887 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.120855    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfn7c\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-kube-api-access-jfn7c\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.121040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.120891    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7178e0-2905-4913-b07f-73b453f0e993-istio-podinfo\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.121040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.120940    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-socket\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.121040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.120971    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-envoy\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.121040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121004    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-certs\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.121040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121030    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-istio-token\") pod \"cc7178e0-2905-4913-b07f-73b453f0e993\" (UID: \"cc7178e0-2905-4913-b07f-73b453f0e993\") "
Jul 18 01:38:02.121326 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121061    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:38:02.121326 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121210    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7178e0-2905-4913-b07f-73b453f0e993-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:38:02.121440 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121321    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:38:02.121440 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121377    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:38:02.121541 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121513    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7178e0-2905-4913-b07f-73b453f0e993-istiod-ca-cert\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.121541 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121528    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-credential-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.121541 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121537    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-socket\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.121741 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121550    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-workload-certs\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.121803 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.121755    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:38:02.123206 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.123180    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-kube-api-access-jfn7c" (OuterVolumeSpecName: "kube-api-access-jfn7c") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "kube-api-access-jfn7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:38:02.123322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.123204    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc7178e0-2905-4913-b07f-73b453f0e993-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 01:38:02.123322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.123219    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:38:02.123322 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.123226    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc7178e0-2905-4913-b07f-73b453f0e993" (UID: "cc7178e0-2905-4913-b07f-73b453f0e993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:38:02.206295 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.206261    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc7178e0-2905-4913-b07f-73b453f0e993" containerID="92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d" exitCode=0
Jul 18 01:38:02.206762 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.206317    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"
Jul 18 01:38:02.206762 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.206347    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" event={"ID":"cc7178e0-2905-4913-b07f-73b453f0e993","Type":"ContainerDied","Data":"92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d"}
Jul 18 01:38:02.206762 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.206400    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr" event={"ID":"cc7178e0-2905-4913-b07f-73b453f0e993","Type":"ContainerDied","Data":"74ae506dfa6c5c9046aa7089d13096666503a196e4f0bdc21517ba1a47ee8909"}
Jul 18 01:38:02.206762 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.206426    2554 scope.go:117] "RemoveContainer" containerID="92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d"
Jul 18 01:38:02.208382 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.208359    2554 generic.go:356] "Generic (PLEG): container finished" podID="f134418f-3d29-496b-8abd-897714961717" containerID="3e37767828476f59c5617cf8266f20f5bec83e1f8a19c481b2f9ce80462875e5" exitCode=6
Jul 18 01:38:02.208506 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.208411    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" event={"ID":"f134418f-3d29-496b-8abd-897714961717","Type":"ContainerDied","Data":"3e37767828476f59c5617cf8266f20f5bec83e1f8a19c481b2f9ce80462875e5"}
Jul 18 01:38:02.208630 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.208603    2554 scope.go:117] "RemoveContainer" containerID="3e37767828476f59c5617cf8266f20f5bec83e1f8a19c481b2f9ce80462875e5"
Jul 18 01:38:02.208847 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:02.208829    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p_odh-ai-gateway-infra(f134418f-3d29-496b-8abd-897714961717)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" podUID="f134418f-3d29-496b-8abd-897714961717"
Jul 18 01:38:02.215128 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.215110    2554 scope.go:117] "RemoveContainer" containerID="57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00"
Jul 18 01:38:02.222450 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.222423    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfn7c\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-kube-api-access-jfn7c\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.222549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.222454    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7178e0-2905-4913-b07f-73b453f0e993-istio-podinfo\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.222549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.222470    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-envoy\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.222549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.222483    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7178e0-2905-4913-b07f-73b453f0e993-istio-token\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.222549 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.222496    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7178e0-2905-4913-b07f-73b453f0e993-istio-data\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:02.226852 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.226835    2554 scope.go:117] "RemoveContainer" containerID="92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d"
Jul 18 01:38:02.227091 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:02.227070    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d\": container with ID starting with 92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d not found: ID does not exist" containerID="92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d"
Jul 18 01:38:02.227171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.227104    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d"} err="failed to get container status \"92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d\": rpc error: code = NotFound desc = could not find container \"92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d\": container with ID starting with 92a2204ab6e77101f55bea090c265b42726efc843164ad865e7ba82cf7cc141d not found: ID does not exist"
Jul 18 01:38:02.227171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.227129    2554 scope.go:117] "RemoveContainer" containerID="57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00"
Jul 18 01:38:02.227390 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:02.227373    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00\": container with ID starting with 57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00 not found: ID does not exist" containerID="57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00"
Jul 18 01:38:02.227434 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.227396    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00"} err="failed to get container status \"57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00\": rpc error: code = NotFound desc = could not find container \"57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00\": container with ID starting with 57557e626a6878c5ca5db0ff2002c72b49252142a8984d4b6a6492d3b6d9ba00 not found: ID does not exist"
Jul 18 01:38:02.227434 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.227411    2554 scope.go:117] "RemoveContainer" containerID="26118cd0e72111a543322a8a7a8a67837941063c476df4be327df282261247e9"
Jul 18 01:38:02.240299 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.240277    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"]
Jul 18 01:38:02.243924 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.243905    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-0f7fdc-openshift-default-7cc6784bf9-bzmdr"]
Jul 18 01:38:02.337444 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:02.337372    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" path="/var/lib/kubelet/pods/cc7178e0-2905-4913-b07f-73b453f0e993/volumes"
Jul 18 01:38:03.123848 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:03.123818    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-mm688_fa3e6edb-9b8c-4b31-a1b6-d234dee87f72/manager/0.log"
Jul 18 01:38:03.129348 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:03.129327    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-mm688_fa3e6edb-9b8c-4b31-a1b6-d234dee87f72/copy-manifests/0.log"
Jul 18 01:38:03.215447 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:03.215419    2554 scope.go:117] "RemoveContainer" containerID="3e37767828476f59c5617cf8266f20f5bec83e1f8a19c481b2f9ce80462875e5"
Jul 18 01:38:03.215916 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:03.215697    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p_odh-ai-gateway-infra(f134418f-3d29-496b-8abd-897714961717)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" podUID="f134418f-3d29-496b-8abd-897714961717"
Jul 18 01:38:03.976691 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:03.976654    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738955-sqz2k_a12b05e5-679d-46b2-be7d-fef5d13239fa/cleanup/0.log"
Jul 18 01:38:04.186229 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:04.186201    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-4701d191-56qnl_30b490bc-772b-4d77-bfe8-f8766ec236ad/revoke-keys/1.log"
Jul 18 01:38:04.288041 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:04.288009    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p_f134418f-3d29-496b-8abd-897714961717/revoke-keys/1.log"
Jul 18 01:38:04.822670 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:04.822642    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-46dd73e1-sljkz_5372c023-1251-414a-8872-693da4c98af0/revoke-keys/0.log"
Jul 18 01:38:04.930032 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:04.930000    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-d1e63f47-xvz7q_c9ca9c53-5ef2-40cd-8053-0622ec37e16d/revoke-keys/0.log"
Jul 18 01:38:05.246162 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:05.246138    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-whmbf_11b679f6-cb84-4f78-a4b9-8b3bdf351301/postgres/0.log"
Jul 18 01:38:07.600963 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.600931    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hf2d6_593c9c0e-8705-409e-a2d4-0546e31fe99d/istio-proxy/0.log"
Jul 18 01:38:07.660902 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.660870    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"]
Jul 18 01:38:07.661284 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.661272    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy"
Jul 18 01:38:07.661328 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.661286    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy"
Jul 18 01:38:07.661328 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.661298    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy"
Jul 18 01:38:07.661328 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.661304    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy"
Jul 18 01:38:07.661428 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.661362    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy"
Jul 18 01:38:07.661428 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.661370    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7178e0-2905-4913-b07f-73b453f0e993" containerName="istio-proxy"
Jul 18 01:38:07.666032 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.666014    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"
Jul 18 01:38:07.671466 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.671441    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"]
Jul 18 01:38:07.713987 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.713960    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ff5sb_6d1510d1-f338-4b0d-a8d8-969029c8dbc4/discovery/0.log"
Jul 18 01:38:07.776938 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.776890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1c381f-b4c5-4653-ae88-93f77e5ee422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n\" (UID: \"8a1c381f-b4c5-4653-ae88-93f77e5ee422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"
Jul 18 01:38:07.878438 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.878334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1c381f-b4c5-4653-ae88-93f77e5ee422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n\" (UID: \"8a1c381f-b4c5-4653-ae88-93f77e5ee422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"
Jul 18 01:38:07.879039 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.879014    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1c381f-b4c5-4653-ae88-93f77e5ee422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n\" (UID: \"8a1c381f-b4c5-4653-ae88-93f77e5ee422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"
Jul 18 01:38:07.978709 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:07.978669    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"
Jul 18 01:38:08.045338 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:08.045309    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rwfcw_a8c9e7b2-5600-4833-96ce-79b2fe2df033/istio-proxy/1.log"
Jul 18 01:38:08.106331 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:08.106306    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"]
Jul 18 01:38:08.108741 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:38:08.108707    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1c381f_b4c5_4653_ae88_93f77e5ee422.slice/crio-59b6f0c847920406a18ed48d73b1510b7484772ae1cb766756be031ed6182cac WatchSource:0}: Error finding container 59b6f0c847920406a18ed48d73b1510b7484772ae1cb766756be031ed6182cac: Status 404 returned error can't find the container with id 59b6f0c847920406a18ed48d73b1510b7484772ae1cb766756be031ed6182cac
Jul 18 01:38:08.165300 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:08.165273    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zvmsp_dfb9430f-32f7-40a0-b155-b06cd7c55f37/payload-pre-processing/0.log"
Jul 18 01:38:08.240985 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:08.240960    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" event={"ID":"8a1c381f-b4c5-4653-ae88-93f77e5ee422","Type":"ContainerStarted","Data":"59b6f0c847920406a18ed48d73b1510b7484772ae1cb766756be031ed6182cac"}
Jul 18 01:38:08.708876 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:08.708788    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wzclz_86a3f4a6-ecd6-48aa-b388-f056ab0fe814/main/0.log"
Jul 18 01:38:08.931847 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:08.931815    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-j9qwv_6c85eddf-8a21-4304-be86-d1314e06f97d/main/0.log"
Jul 18 01:38:09.247633 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:09.247586    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a1c381f-b4c5-4653-ae88-93f77e5ee422" containerID="f7f5d9e26ffc272de08c111ed319d852e3c205914720e2cb876b874ac3f89540" exitCode=6
Jul 18 01:38:09.247845 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:09.247677    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" event={"ID":"8a1c381f-b4c5-4653-ae88-93f77e5ee422","Type":"ContainerDied","Data":"f7f5d9e26ffc272de08c111ed319d852e3c205914720e2cb876b874ac3f89540"}
Jul 18 01:38:09.247923 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:09.247888    2554 scope.go:117] "RemoveContainer" containerID="f7f5d9e26ffc272de08c111ed319d852e3c205914720e2cb876b874ac3f89540"
Jul 18 01:38:10.254957 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:10.254923    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a1c381f-b4c5-4653-ae88-93f77e5ee422" containerID="89879ef1ee029b17ec8a41e96dad5ea67c45b321d2148be061ff14804c5aedca" exitCode=6
Jul 18 01:38:10.255369 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:10.255013    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" event={"ID":"8a1c381f-b4c5-4653-ae88-93f77e5ee422","Type":"ContainerDied","Data":"89879ef1ee029b17ec8a41e96dad5ea67c45b321d2148be061ff14804c5aedca"}
Jul 18 01:38:10.255369 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:10.255074    2554 scope.go:117] "RemoveContainer" containerID="f7f5d9e26ffc272de08c111ed319d852e3c205914720e2cb876b874ac3f89540"
Jul 18 01:38:10.255369 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:10.255171    2554 scope.go:117] "RemoveContainer" containerID="89879ef1ee029b17ec8a41e96dad5ea67c45b321d2148be061ff14804c5aedca"
Jul 18 01:38:10.255495 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:10.255450    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n_odh-ai-gateway-infra(8a1c381f-b4c5-4653-ae88-93f77e5ee422)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" podUID="8a1c381f-b4c5-4653-ae88-93f77e5ee422"
Jul 18 01:38:11.261093 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:11.261058    2554 scope.go:117] "RemoveContainer" containerID="89879ef1ee029b17ec8a41e96dad5ea67c45b321d2148be061ff14804c5aedca"
Jul 18 01:38:11.261468 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:11.261284    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n_odh-ai-gateway-infra(8a1c381f-b4c5-4653-ae88-93f77e5ee422)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" podUID="8a1c381f-b4c5-4653-ae88-93f77e5ee422"
Jul 18 01:38:12.330441 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:12.330414    2554 scope.go:117] "RemoveContainer" containerID="fb8a02a14c6eca35922421d05ae7128ebc7366d21d35f70eb2073a9662ecc008"
Jul 18 01:38:13.272818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:13.272734    2554 generic.go:356] "Generic (PLEG): container finished" podID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerID="7d7366f34df825f6d867b984427df17eff3f31ac5957e66ceb8fdf29c6324388" exitCode=6
Jul 18 01:38:13.272818 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:13.272796    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" event={"ID":"30b490bc-772b-4d77-bfe8-f8766ec236ad","Type":"ContainerDied","Data":"7d7366f34df825f6d867b984427df17eff3f31ac5957e66ceb8fdf29c6324388"}
Jul 18 01:38:13.273025 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:13.272846    2554 scope.go:117] "RemoveContainer" containerID="fb8a02a14c6eca35922421d05ae7128ebc7366d21d35f70eb2073a9662ecc008"
Jul 18 01:38:13.273185 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:13.273167    2554 scope.go:117] "RemoveContainer" containerID="7d7366f34df825f6d867b984427df17eff3f31ac5957e66ceb8fdf29c6324388"
Jul 18 01:38:13.273430 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:13.273413    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-56qnl_odh-ai-gateway-infra(30b490bc-772b-4d77-bfe8-f8766ec236ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad"
Jul 18 01:38:14.297744 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:14.297711    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"]
Jul 18 01:38:14.435076 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:14.435048    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"
Jul 18 01:38:14.546911 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:14.546878    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b490bc-772b-4d77-bfe8-f8766ec236ad-maas-api-service-ca\") pod \"30b490bc-772b-4d77-bfe8-f8766ec236ad\" (UID: \"30b490bc-772b-4d77-bfe8-f8766ec236ad\") "
Jul 18 01:38:14.547220 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:14.547200    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b490bc-772b-4d77-bfe8-f8766ec236ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30b490bc-772b-4d77-bfe8-f8766ec236ad" (UID: "30b490bc-772b-4d77-bfe8-f8766ec236ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:38:14.648504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:14.648420    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b490bc-772b-4d77-bfe8-f8766ec236ad-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:15.285026 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:15.284997    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"
Jul 18 01:38:15.285225 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:15.284993    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl" event={"ID":"30b490bc-772b-4d77-bfe8-f8766ec236ad","Type":"ContainerDied","Data":"a6c4a3a6337d87e6f8256fc3595ddb496085664d507d0da61959899c4ff68b8e"}
Jul 18 01:38:15.285225 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:15.285174    2554 scope.go:117] "RemoveContainer" containerID="7d7366f34df825f6d867b984427df17eff3f31ac5957e66ceb8fdf29c6324388"
Jul 18 01:38:15.308477 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:15.308450    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"]
Jul 18 01:38:15.314215 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:15.314194    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-56qnl"]
Jul 18 01:38:16.330605 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:16.330573    2554 scope.go:117] "RemoveContainer" containerID="3e37767828476f59c5617cf8266f20f5bec83e1f8a19c481b2f9ce80462875e5"
Jul 18 01:38:16.336583 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:16.336557    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" path="/var/lib/kubelet/pods/30b490bc-772b-4d77-bfe8-f8766ec236ad/volumes"
Jul 18 01:38:17.298054 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:17.298018    2554 generic.go:356] "Generic (PLEG): container finished" podID="f134418f-3d29-496b-8abd-897714961717" containerID="2ccc97c7d08d7f4d209a3cf6fdc1a84cfd417a3800c7a64b83ad688b8aba7c27" exitCode=6
Jul 18 01:38:17.298252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:17.298094    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" event={"ID":"f134418f-3d29-496b-8abd-897714961717","Type":"ContainerDied","Data":"2ccc97c7d08d7f4d209a3cf6fdc1a84cfd417a3800c7a64b83ad688b8aba7c27"}
Jul 18 01:38:17.298252 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:17.298162    2554 scope.go:117] "RemoveContainer" containerID="3e37767828476f59c5617cf8266f20f5bec83e1f8a19c481b2f9ce80462875e5"
Jul 18 01:38:17.298531 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:17.298511    2554 scope.go:117] "RemoveContainer" containerID="2ccc97c7d08d7f4d209a3cf6fdc1a84cfd417a3800c7a64b83ad688b8aba7c27"
Jul 18 01:38:17.298805 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:17.298785    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p_odh-ai-gateway-infra(f134418f-3d29-496b-8abd-897714961717)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" podUID="f134418f-3d29-496b-8abd-897714961717"
Jul 18 01:38:17.477983 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:17.477956    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tc4gd_a089e7e1-1597-4899-83b2-357a35bb40cb/global-pull-secret-syncer/0.log"
Jul 18 01:38:17.653504 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:17.653430    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t96fg_0621ebeb-5101-4564-9c03-a208a0a105f7/konnectivity-agent/0.log"
Jul 18 01:38:17.694663 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:17.694630    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-161.ec2.internal_6a0ab874ed9de1072b24764a5ad02232/haproxy/0.log"
Jul 18 01:38:18.326053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:18.326023    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"]
Jul 18 01:38:18.479704 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:18.479676    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"
Jul 18 01:38:18.586416 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:18.586312    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f134418f-3d29-496b-8abd-897714961717-maas-api-service-ca\") pod \"f134418f-3d29-496b-8abd-897714961717\" (UID: \"f134418f-3d29-496b-8abd-897714961717\") "
Jul 18 01:38:18.586742 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:18.586717    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f134418f-3d29-496b-8abd-897714961717-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f134418f-3d29-496b-8abd-897714961717" (UID: "f134418f-3d29-496b-8abd-897714961717"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:38:18.687729 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:18.687689    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f134418f-3d29-496b-8abd-897714961717-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:19.310175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:19.310142    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"
Jul 18 01:38:19.310175 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:19.310167    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p" event={"ID":"f134418f-3d29-496b-8abd-897714961717","Type":"ContainerDied","Data":"46579724b8829616584c103cd7a246f9d00242b51fd9747ebfa0aecd93826751"}
Jul 18 01:38:19.310426 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:19.310222    2554 scope.go:117] "RemoveContainer" containerID="2ccc97c7d08d7f4d209a3cf6fdc1a84cfd417a3800c7a64b83ad688b8aba7c27"
Jul 18 01:38:19.332251 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:19.332226    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"]
Jul 18 01:38:19.337857 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:19.337836    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-c7v9p"]
Jul 18 01:38:20.335461 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:20.335428    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f134418f-3d29-496b-8abd-897714961717" path="/var/lib/kubelet/pods/f134418f-3d29-496b-8abd-897714961717/volumes"
Jul 18 01:38:22.737785 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.737731    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"]
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738142    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738155    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738162    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738167    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738175    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738181    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738188    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738193    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738210    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738214    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738268    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738277    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738286    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738293    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:22.738298 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.738298    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:22.743325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.743301    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"
Jul 18 01:38:22.748563 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.748536    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"]
Jul 18 01:38:22.830660 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.830605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0460ab4e-098f-49e6-866c-6cce2d62ff5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch\" (UID: \"0460ab4e-098f-49e6-866c-6cce2d62ff5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"
Jul 18 01:38:22.931975 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.931948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0460ab4e-098f-49e6-866c-6cce2d62ff5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch\" (UID: \"0460ab4e-098f-49e6-866c-6cce2d62ff5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"
Jul 18 01:38:22.932546 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:22.932526    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0460ab4e-098f-49e6-866c-6cce2d62ff5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch\" (UID: \"0460ab4e-098f-49e6-866c-6cce2d62ff5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"
Jul 18 01:38:23.055541 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:23.055505    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"
Jul 18 01:38:23.188715 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:23.188690    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"]
Jul 18 01:38:23.191032 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:38:23.191004    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0460ab4e_098f_49e6_866c_6cce2d62ff5a.slice/crio-71f97b7e0068513d4010260e65c0604ab7059cf3b2e76873160dcc0edb540f4f WatchSource:0}: Error finding container 71f97b7e0068513d4010260e65c0604ab7059cf3b2e76873160dcc0edb540f4f: Status 404 returned error can't find the container with id 71f97b7e0068513d4010260e65c0604ab7059cf3b2e76873160dcc0edb540f4f
Jul 18 01:38:23.331900 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:23.331817    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" event={"ID":"0460ab4e-098f-49e6-866c-6cce2d62ff5a","Type":"ContainerStarted","Data":"71f97b7e0068513d4010260e65c0604ab7059cf3b2e76873160dcc0edb540f4f"}
Jul 18 01:38:24.337908 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:24.337870    2554 generic.go:356] "Generic (PLEG): container finished" podID="0460ab4e-098f-49e6-866c-6cce2d62ff5a" containerID="8e61986ae6e558b78131e208a16faa6083ea66affc19f36ce5accbe453302d25" exitCode=6
Jul 18 01:38:24.338292 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:24.337930    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" event={"ID":"0460ab4e-098f-49e6-866c-6cce2d62ff5a","Type":"ContainerDied","Data":"8e61986ae6e558b78131e208a16faa6083ea66affc19f36ce5accbe453302d25"}
Jul 18 01:38:24.338292 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:24.338082    2554 scope.go:117] "RemoveContainer" containerID="8e61986ae6e558b78131e208a16faa6083ea66affc19f36ce5accbe453302d25"
Jul 18 01:38:25.346961 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.346932    2554 generic.go:356] "Generic (PLEG): container finished" podID="0460ab4e-098f-49e6-866c-6cce2d62ff5a" containerID="a73c36ef6731c9cc1944d8f80db8016b82f551b4132c692ea051779e3aa7e86e" exitCode=6
Jul 18 01:38:25.347354 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.346998    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" event={"ID":"0460ab4e-098f-49e6-866c-6cce2d62ff5a","Type":"ContainerDied","Data":"a73c36ef6731c9cc1944d8f80db8016b82f551b4132c692ea051779e3aa7e86e"}
Jul 18 01:38:25.347354 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.347068    2554 scope.go:117] "RemoveContainer" containerID="8e61986ae6e558b78131e208a16faa6083ea66affc19f36ce5accbe453302d25"
Jul 18 01:38:25.347354 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.347230    2554 scope.go:117] "RemoveContainer" containerID="a73c36ef6731c9cc1944d8f80db8016b82f551b4132c692ea051779e3aa7e86e"
Jul 18 01:38:25.347495 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:25.347472    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch_odh-ai-gateway-infra(0460ab4e-098f-49e6-866c-6cce2d62ff5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" podUID="0460ab4e-098f-49e6-866c-6cce2d62ff5a"
Jul 18 01:38:25.592189 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.592159    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a4a8b40e-7a11-482f-b9fb-27cab2d81aa8/alertmanager/0.log"
Jul 18 01:38:25.616167 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.616100    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a4a8b40e-7a11-482f-b9fb-27cab2d81aa8/config-reloader/0.log"
Jul 18 01:38:25.637608 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.637587    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a4a8b40e-7a11-482f-b9fb-27cab2d81aa8/kube-rbac-proxy-web/0.log"
Jul 18 01:38:25.657731 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.657708    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a4a8b40e-7a11-482f-b9fb-27cab2d81aa8/kube-rbac-proxy/0.log"
Jul 18 01:38:25.678393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.678366    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a4a8b40e-7a11-482f-b9fb-27cab2d81aa8/kube-rbac-proxy-metric/0.log"
Jul 18 01:38:25.698168 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.698149    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a4a8b40e-7a11-482f-b9fb-27cab2d81aa8/prom-label-proxy/0.log"
Jul 18 01:38:25.718108 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.718088    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a4a8b40e-7a11-482f-b9fb-27cab2d81aa8/init-config-reloader/0.log"
Jul 18 01:38:25.800922 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.800900    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-x744k_8c77e01c-9265-452d-b94d-30c8eb6a6b56/kube-state-metrics/0.log"
Jul 18 01:38:25.820458 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.820436    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-x744k_8c77e01c-9265-452d-b94d-30c8eb6a6b56/kube-rbac-proxy-main/0.log"
Jul 18 01:38:25.839601 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.839579    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-x744k_8c77e01c-9265-452d-b94d-30c8eb6a6b56/kube-rbac-proxy-self/0.log"
Jul 18 01:38:25.866383 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.866329    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c5ccd67dc-wk5wk_a2040fa1-7916-4610-bfcb-445d2faa646e/metrics-server/0.log"
Jul 18 01:38:25.889750 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.889728    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-8wj2j_f7e8eece-45ff-4481-b41a-bad92ad7fdad/monitoring-plugin/0.log"
Jul 18 01:38:25.915180 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.915158    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mtpf_6e282b8c-73e9-491f-bfa5-46be537dd4d4/node-exporter/0.log"
Jul 18 01:38:25.933521 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.933493    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mtpf_6e282b8c-73e9-491f-bfa5-46be537dd4d4/kube-rbac-proxy/0.log"
Jul 18 01:38:25.952831 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:25.952809    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mtpf_6e282b8c-73e9-491f-bfa5-46be537dd4d4/init-textfile/0.log"
Jul 18 01:38:26.112409 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.112379    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-tllg7_e284174e-dce4-4f66-8f8c-c58ae7038721/kube-rbac-proxy-main/0.log"
Jul 18 01:38:26.133962 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.133893    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-tllg7_e284174e-dce4-4f66-8f8c-c58ae7038721/kube-rbac-proxy-self/0.log"
Jul 18 01:38:26.153042 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.153017    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-tllg7_e284174e-dce4-4f66-8f8c-c58ae7038721/openshift-state-metrics/0.log"
Jul 18 01:38:26.189701 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.189678    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e6b714e1-8f8d-482a-93f4-71488ffd7269/prometheus/0.log"
Jul 18 01:38:26.208380 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.208360    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e6b714e1-8f8d-482a-93f4-71488ffd7269/config-reloader/0.log"
Jul 18 01:38:26.228383 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.228359    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e6b714e1-8f8d-482a-93f4-71488ffd7269/thanos-sidecar/0.log"
Jul 18 01:38:26.261287 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.261265    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e6b714e1-8f8d-482a-93f4-71488ffd7269/kube-rbac-proxy-web/0.log"
Jul 18 01:38:26.281581 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.281562    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e6b714e1-8f8d-482a-93f4-71488ffd7269/kube-rbac-proxy/0.log"
Jul 18 01:38:26.299976 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.299959    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e6b714e1-8f8d-482a-93f4-71488ffd7269/kube-rbac-proxy-thanos/0.log"
Jul 18 01:38:26.324263 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.324244    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e6b714e1-8f8d-482a-93f4-71488ffd7269/init-config-reloader/0.log"
Jul 18 01:38:26.331072 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.331056    2554 scope.go:117] "RemoveContainer" containerID="89879ef1ee029b17ec8a41e96dad5ea67c45b321d2148be061ff14804c5aedca"
Jul 18 01:38:26.349228 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.349193    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-btpwm_2dea8cd9-8a03-4db8-9d07-230fcb4ca17e/prometheus-operator/0.log"
Jul 18 01:38:26.353895 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.353876    2554 scope.go:117] "RemoveContainer" containerID="a73c36ef6731c9cc1944d8f80db8016b82f551b4132c692ea051779e3aa7e86e"
Jul 18 01:38:26.354121 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:26.354103    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch_odh-ai-gateway-infra(0460ab4e-098f-49e6-866c-6cce2d62ff5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" podUID="0460ab4e-098f-49e6-866c-6cce2d62ff5a"
Jul 18 01:38:26.369053 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.369034    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-btpwm_2dea8cd9-8a03-4db8-9d07-230fcb4ca17e/kube-rbac-proxy/0.log"
Jul 18 01:38:26.390297 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.390248    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-v5swd_069929f9-0ece-4af0-a944-5cba5faaf972/prometheus-operator-admission-webhook/0.log"
Jul 18 01:38:26.482829 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.482803    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b987bb95b-mctr7_17809a4e-df1b-40b8-9b2f-10f489d32b97/thanos-query/0.log"
Jul 18 01:38:26.501075 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.501052    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b987bb95b-mctr7_17809a4e-df1b-40b8-9b2f-10f489d32b97/kube-rbac-proxy-web/0.log"
Jul 18 01:38:26.537554 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.537537    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b987bb95b-mctr7_17809a4e-df1b-40b8-9b2f-10f489d32b97/kube-rbac-proxy/0.log"
Jul 18 01:38:26.557171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.557149    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b987bb95b-mctr7_17809a4e-df1b-40b8-9b2f-10f489d32b97/prom-label-proxy/0.log"
Jul 18 01:38:26.576756 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.576734    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b987bb95b-mctr7_17809a4e-df1b-40b8-9b2f-10f489d32b97/kube-rbac-proxy-rules/0.log"
Jul 18 01:38:26.595325 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:26.595302    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b987bb95b-mctr7_17809a4e-df1b-40b8-9b2f-10f489d32b97/kube-rbac-proxy-metrics/0.log"
Jul 18 01:38:27.360103 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.360066    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a1c381f-b4c5-4653-ae88-93f77e5ee422" containerID="0a4a5fa899a11aa2e925bd4e21f485457fea22276bdc5c976f1256538119b877" exitCode=6
Jul 18 01:38:27.360596 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.360135    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" event={"ID":"8a1c381f-b4c5-4653-ae88-93f77e5ee422","Type":"ContainerDied","Data":"0a4a5fa899a11aa2e925bd4e21f485457fea22276bdc5c976f1256538119b877"}
Jul 18 01:38:27.360596 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.360194    2554 scope.go:117] "RemoveContainer" containerID="89879ef1ee029b17ec8a41e96dad5ea67c45b321d2148be061ff14804c5aedca"
Jul 18 01:38:27.360753 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.360600    2554 scope.go:117] "RemoveContainer" containerID="0a4a5fa899a11aa2e925bd4e21f485457fea22276bdc5c976f1256538119b877"
Jul 18 01:38:27.360905 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:27.360878    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n_odh-ai-gateway-infra(8a1c381f-b4c5-4653-ae88-93f77e5ee422)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" podUID="8a1c381f-b4c5-4653-ae88-93f77e5ee422"
Jul 18 01:38:27.388631 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.388585    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"]
Jul 18 01:38:27.389004 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.388991    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:27.389048 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.389006    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f134418f-3d29-496b-8abd-897714961717" containerName="revoke-keys"
Jul 18 01:38:27.389086 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.389076    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b490bc-772b-4d77-bfe8-f8766ec236ad" containerName="revoke-keys"
Jul 18 01:38:27.393224 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.393206    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.395576 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.395553    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gb6px\"/\"openshift-service-ca.crt\""
Jul 18 01:38:27.395697 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.395596    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gb6px\"/\"default-dockercfg-hr289\""
Jul 18 01:38:27.395697 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.395596    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gb6px\"/\"kube-root-ca.crt\""
Jul 18 01:38:27.402077 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.402054    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"]
Jul 18 01:38:27.579111 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.579083    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqn8r\" (UniqueName: \"kubernetes.io/projected/facafda6-160b-4dad-a972-4f925e7dc5c8-kube-api-access-nqn8r\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.579304 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.579134    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-sys\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.579304 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.579189    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-podres\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.579304 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.579221    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-lib-modules\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.579415 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.579370    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-proc\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-sys\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680448    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-podres\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680491 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680482    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-lib-modules\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680747 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680532    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-sys\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680747 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-proc\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680747 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680585    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-podres\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680747 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680593    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqn8r\" (UniqueName: \"kubernetes.io/projected/facafda6-160b-4dad-a972-4f925e7dc5c8-kube-api-access-nqn8r\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680747 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680658    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-proc\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.680747 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.680663    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facafda6-160b-4dad-a972-4f925e7dc5c8-lib-modules\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.688022 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.688002    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqn8r\" (UniqueName: \"kubernetes.io/projected/facafda6-160b-4dad-a972-4f925e7dc5c8-kube-api-access-nqn8r\") pod \"perf-node-gather-daemonset-h7ddx\" (UID: \"facafda6-160b-4dad-a972-4f925e7dc5c8\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.703943 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.703922    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:27.831792 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:27.831764    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"]
Jul 18 01:38:27.833491 ip-10-0-141-161 kubenswrapper[2554]: W0718 01:38:27.833462    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfacafda6_160b_4dad_a972_4f925e7dc5c8.slice/crio-24e3667eaeed9273795390f5e3ad37fb497174d64cac40403920cfdb0ae84dce WatchSource:0}: Error finding container 24e3667eaeed9273795390f5e3ad37fb497174d64cac40403920cfdb0ae84dce: Status 404 returned error can't find the container with id 24e3667eaeed9273795390f5e3ad37fb497174d64cac40403920cfdb0ae84dce
Jul 18 01:38:28.368632 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.368583    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx" event={"ID":"facafda6-160b-4dad-a972-4f925e7dc5c8","Type":"ContainerStarted","Data":"d11946b2c861ba50a4e9dcc65479144d11c2847d8d9afe70794b339a145f25e5"}
Jul 18 01:38:28.368632 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.368649    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx" event={"ID":"facafda6-160b-4dad-a972-4f925e7dc5c8","Type":"ContainerStarted","Data":"24e3667eaeed9273795390f5e3ad37fb497174d64cac40403920cfdb0ae84dce"}
Jul 18 01:38:28.369171 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.368671    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:28.385580 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.385541    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"]
Jul 18 01:38:28.386988 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.386934    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx" podStartSLOduration=1.386915749 podStartE2EDuration="1.386915749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:38:28.385702365 +0000 UTC m=+3464.511637344" watchObservedRunningTime="2026-07-18 01:38:28.386915749 +0000 UTC m=+3464.512850730"
Jul 18 01:38:28.503282 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.503257    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"
Jul 18 01:38:28.689784 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.689705    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1c381f-b4c5-4653-ae88-93f77e5ee422-maas-api-service-ca\") pod \"8a1c381f-b4c5-4653-ae88-93f77e5ee422\" (UID: \"8a1c381f-b4c5-4653-ae88-93f77e5ee422\") "
Jul 18 01:38:28.690108 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.690076    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1c381f-b4c5-4653-ae88-93f77e5ee422-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a1c381f-b4c5-4653-ae88-93f77e5ee422" (UID: "8a1c381f-b4c5-4653-ae88-93f77e5ee422"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:38:28.790863 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:28.790822    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1c381f-b4c5-4653-ae88-93f77e5ee422-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:29.376007 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:29.375975    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"
Jul 18 01:38:29.376484 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:29.376025    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n" event={"ID":"8a1c381f-b4c5-4653-ae88-93f77e5ee422","Type":"ContainerDied","Data":"59b6f0c847920406a18ed48d73b1510b7484772ae1cb766756be031ed6182cac"}
Jul 18 01:38:29.376484 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:29.376087    2554 scope.go:117] "RemoveContainer" containerID="0a4a5fa899a11aa2e925bd4e21f485457fea22276bdc5c976f1256538119b877"
Jul 18 01:38:29.403939 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:29.403913    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"]
Jul 18 01:38:29.406642 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:29.406604    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v987n"]
Jul 18 01:38:30.337116 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:30.337081    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1c381f-b4c5-4653-ae88-93f77e5ee422" path="/var/lib/kubelet/pods/8a1c381f-b4c5-4653-ae88-93f77e5ee422/volumes"
Jul 18 01:38:30.502082 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:30.502054    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kw5b5_b63d0c20-aae3-43de-a175-20b8fb23fd1c/dns/0.log"
Jul 18 01:38:30.527244 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:30.527217    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kw5b5_b63d0c20-aae3-43de-a175-20b8fb23fd1c/kube-rbac-proxy/0.log"
Jul 18 01:38:30.552319 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:30.552294    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9jvs2_794706ac-dc6b-451e-9dfc-688409907cf9/dns-node-resolver/0.log"
Jul 18 01:38:31.206040 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:31.206006    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tjbks_482bb2e3-16af-4029-b9cb-4ce88e7cc092/node-ca/0.log"
Jul 18 01:38:32.172229 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:32.172195    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hf2d6_593c9c0e-8705-409e-a2d4-0546e31fe99d/istio-proxy/0.log"
Jul 18 01:38:32.289007 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:32.288974    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ff5sb_6d1510d1-f338-4b0d-a8d8-969029c8dbc4/discovery/0.log"
Jul 18 01:38:32.361428 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:32.361397    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rwfcw_a8c9e7b2-5600-4833-96ce-79b2fe2df033/istio-proxy/0.log"
Jul 18 01:38:32.389826 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:32.389800    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rwfcw_a8c9e7b2-5600-4833-96ce-79b2fe2df033/istio-proxy/1.log"
Jul 18 01:38:32.416589 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:32.416561    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zvmsp_dfb9430f-32f7-40a0-b155-b06cd7c55f37/payload-pre-processing/0.log"
Jul 18 01:38:33.172079 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:33.172047    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k28lr_9d36b8cf-0af7-46ad-a1d3-b4456fbba020/serve-healthcheck-canary/0.log"
Jul 18 01:38:33.751423 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:33.751396    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tccd_55e9d85e-ba32-4a40-b6d5-b9931ecbd044/kube-rbac-proxy/0.log"
Jul 18 01:38:33.771050 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:33.771024    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tccd_55e9d85e-ba32-4a40-b6d5-b9931ecbd044/exporter/0.log"
Jul 18 01:38:33.790047 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:33.790023    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tccd_55e9d85e-ba32-4a40-b6d5-b9931ecbd044/extractor/0.log"
Jul 18 01:38:34.386275 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:34.386250    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-h7ddx"
Jul 18 01:38:36.730324 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:36.730293    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-mm688_fa3e6edb-9b8c-4b31-a1b6-d234dee87f72/manager/0.log"
Jul 18 01:38:36.753427 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:36.753404    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-mm688_fa3e6edb-9b8c-4b31-a1b6-d234dee87f72/copy-manifests/0.log"
Jul 18 01:38:38.301153 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:38.301113    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hf2d6_593c9c0e-8705-409e-a2d4-0546e31fe99d/istio-proxy/0.log"
Jul 18 01:38:38.317013 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:38.316981    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b88d5d556-td8hq_5644a48d-2976-4b6f-b410-384b1bf72056/manager/0.log"
Jul 18 01:38:38.411261 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:38.411232    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ff5sb_6d1510d1-f338-4b0d-a8d8-969029c8dbc4/discovery/0.log"
Jul 18 01:38:38.468488 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:38.468458    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rwfcw_a8c9e7b2-5600-4833-96ce-79b2fe2df033/istio-proxy/0.log"
Jul 18 01:38:38.498290 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:38.498269    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rwfcw_a8c9e7b2-5600-4833-96ce-79b2fe2df033/istio-proxy/1.log"
Jul 18 01:38:38.516181 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:38.516155    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zvmsp_dfb9430f-32f7-40a0-b155-b06cd7c55f37/payload-pre-processing/0.log"
Jul 18 01:38:39.330449 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:39.330372    2554 scope.go:117] "RemoveContainer" containerID="a73c36ef6731c9cc1944d8f80db8016b82f551b4132c692ea051779e3aa7e86e"
Jul 18 01:38:39.769699 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:39.769675    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hf2d6_593c9c0e-8705-409e-a2d4-0546e31fe99d/istio-proxy/0.log"
Jul 18 01:38:39.877843 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:39.877811    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ff5sb_6d1510d1-f338-4b0d-a8d8-969029c8dbc4/discovery/0.log"
Jul 18 01:38:39.932636 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:39.932585    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rwfcw_a8c9e7b2-5600-4833-96ce-79b2fe2df033/istio-proxy/0.log"
Jul 18 01:38:39.962393 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:39.962362    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rwfcw_a8c9e7b2-5600-4833-96ce-79b2fe2df033/istio-proxy/1.log"
Jul 18 01:38:39.982283 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:39.982257    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-zvmsp_dfb9430f-32f7-40a0-b155-b06cd7c55f37/payload-pre-processing/0.log"
Jul 18 01:38:40.431124 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:40.431090    2554 generic.go:356] "Generic (PLEG): container finished" podID="0460ab4e-098f-49e6-866c-6cce2d62ff5a" containerID="913ca0dedacbaf55bcc6afa42fbea2094692d472fb7798415cc42dfe82b395b2" exitCode=6
Jul 18 01:38:40.431552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:40.431163    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" event={"ID":"0460ab4e-098f-49e6-866c-6cce2d62ff5a","Type":"ContainerDied","Data":"913ca0dedacbaf55bcc6afa42fbea2094692d472fb7798415cc42dfe82b395b2"}
Jul 18 01:38:40.431552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:40.431228    2554 scope.go:117] "RemoveContainer" containerID="a73c36ef6731c9cc1944d8f80db8016b82f551b4132c692ea051779e3aa7e86e"
Jul 18 01:38:40.431552 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:40.431455    2554 scope.go:117] "RemoveContainer" containerID="913ca0dedacbaf55bcc6afa42fbea2094692d472fb7798415cc42dfe82b395b2"
Jul 18 01:38:40.431760 ip-10-0-141-161 kubenswrapper[2554]: E0718 01:38:40.431740    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch_odh-ai-gateway-infra(0460ab4e-098f-49e6-866c-6cce2d62ff5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" podUID="0460ab4e-098f-49e6-866c-6cce2d62ff5a"
Jul 18 01:38:41.455470 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:41.455440    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"]
Jul 18 01:38:41.599952 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:41.599925    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"
Jul 18 01:38:41.711405 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:41.711308    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0460ab4e-098f-49e6-866c-6cce2d62ff5a-maas-api-service-ca\") pod \"0460ab4e-098f-49e6-866c-6cce2d62ff5a\" (UID: \"0460ab4e-098f-49e6-866c-6cce2d62ff5a\") "
Jul 18 01:38:41.711881 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:41.711846    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0460ab4e-098f-49e6-866c-6cce2d62ff5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0460ab4e-098f-49e6-866c-6cce2d62ff5a" (UID: "0460ab4e-098f-49e6-866c-6cce2d62ff5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:38:41.812329 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:41.812296    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0460ab4e-098f-49e6-866c-6cce2d62ff5a-maas-api-service-ca\") on node \"ip-10-0-141-161.ec2.internal\" DevicePath \"\""
Jul 18 01:38:42.443669 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:42.443636    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"
Jul 18 01:38:42.443843 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:42.443642    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch" event={"ID":"0460ab4e-098f-49e6-866c-6cce2d62ff5a","Type":"ContainerDied","Data":"71f97b7e0068513d4010260e65c0604ab7059cf3b2e76873160dcc0edb540f4f"}
Jul 18 01:38:42.443843 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:42.443774    2554 scope.go:117] "RemoveContainer" containerID="913ca0dedacbaf55bcc6afa42fbea2094692d472fb7798415cc42dfe82b395b2"
Jul 18 01:38:42.459248 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:42.459224    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"]
Jul 18 01:38:42.463894 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:42.463870    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xvxch"]
Jul 18 01:38:44.336572 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.336541    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0460ab4e-098f-49e6-866c-6cce2d62ff5a" path="/var/lib/kubelet/pods/0460ab4e-098f-49e6-866c-6cce2d62ff5a/volumes"
Jul 18 01:38:44.679564 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.679475    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-546f9_761c310d-2fe7-4644-aec4-2182615102af/kube-multus/0.log"
Jul 18 01:38:44.782485 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.782458    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9k5md_9b2c4b45-013f-4e4a-8e8c-a73b0bba4498/kube-multus-additional-cni-plugins/0.log"
Jul 18 01:38:44.805718 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.805694    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9k5md_9b2c4b45-013f-4e4a-8e8c-a73b0bba4498/egress-router-binary-copy/0.log"
Jul 18 01:38:44.826197 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.826169    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9k5md_9b2c4b45-013f-4e4a-8e8c-a73b0bba4498/cni-plugins/0.log"
Jul 18 01:38:44.847125 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.847107    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9k5md_9b2c4b45-013f-4e4a-8e8c-a73b0bba4498/bond-cni-plugin/0.log"
Jul 18 01:38:44.871183 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.871158    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9k5md_9b2c4b45-013f-4e4a-8e8c-a73b0bba4498/routeoverride-cni/0.log"
Jul 18 01:38:44.892512 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.892484    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9k5md_9b2c4b45-013f-4e4a-8e8c-a73b0bba4498/whereabouts-cni-bincopy/0.log"
Jul 18 01:38:44.913112 ip-10-0-141-161 kubenswrapper[2554]: I0718 01:38:44.913084    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9k5md_9b2c4b45-013f-4e4a-8e8c-a73b0bba4498/whereabouts-cni/0.log"