Sep 09 17:17:02.920261 ip-10-0-139-100 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 17:17:03.237586 ip-10-0-139-100 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 17:17:03.237586 ip-10-0-139-100 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 17:17:03.237586 ip-10-0-139-100 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 17:17:03.237586 ip-10-0-139-100 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 17:17:03.237586 ip-10-0-139-100 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 17:17:03.239556 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.239480    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 17:17:03.242793 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242777    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 17:17:03.242793 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242792    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242796    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242802    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242806    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242810    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242813    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242816    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242819    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242822    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242825    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242827    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242830    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242833    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242837    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242840    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242843    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242846    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242850    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242852    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242855    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 17:17:03.242860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242858    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242860    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242863    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242866    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242869    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242872    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242875    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242878    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242880    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242883    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242886    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242889    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242892    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242894    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242897    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242900    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242902    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242905    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242908    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242910    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 17:17:03.243431 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242913    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242915    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242918    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242921    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242924    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242926    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242928    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242931    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242934    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242936    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242939    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242941    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242944    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242947    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242950    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242953    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242956    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242958    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242961    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 17:17:03.243912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242964    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242966    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242969    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242971    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242974    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242976    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242979    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242981    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242984    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.242999    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243003    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243006    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243009    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243012    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243014    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243017    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243020    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243022    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243025    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243027    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 17:17:03.244378 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243030    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243033    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243035    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243038    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243041    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243044    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243047    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243050    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243054    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243058    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243062    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243065    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243068    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243071    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243074    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243077    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243080    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243082    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243085    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 17:17:03.244860 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243767    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243773    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243776    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243779    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243782    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243786    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243789    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243791    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243794    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243797    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243800    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243802    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243805    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243807    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243810    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243812    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243815    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243819    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243823    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 17:17:03.245355 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243826    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243831    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243834    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243837    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243840    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243843    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243845    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243848    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243852    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243855    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243857    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243860    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243863    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243865    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243868    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243871    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243873    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243876    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243878    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243882    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 17:17:03.245830 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243884    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243886    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243889    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243891    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243894    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243896    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243898    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243901    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243903    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243906    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243909    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243911    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243914    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243917    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243920    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243923    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243925    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243928    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243930    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 17:17:03.246341 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243933    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243935    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243938    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243940    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243943    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243945    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243947    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243950    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243952    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243955    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243957    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243959    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243963    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243972    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243975    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243977    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243979    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243982    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.243985    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 17:17:03.246811 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244001    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244003    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244006    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244009    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244011    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244014    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244016    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244019    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244022    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244026    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244029    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244032    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244035    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244037    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244040    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244042    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244045    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244047    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244050    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244053    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 17:17:03.247287 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244055    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244058    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244420    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244425    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244428    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244431    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244434    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244437    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244440    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244443    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244446    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244448    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244451    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244453    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244456    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244459    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244461    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244464    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244467    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244470    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 17:17:03.247768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244473    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244476    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244478    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244481    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244483    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244486    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244489    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244491    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244494    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244496    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244499    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244501    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244504    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244506    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244509    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244511    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244514    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244516    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244518    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244521    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 17:17:03.248279 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244524    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244527    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244529    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244532    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244534    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244537    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244539    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244542    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244545    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244547    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244551    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244553    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244556    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244558    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244561    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244563    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244566    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244568    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244571    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244573    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 17:17:03.248768 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244576    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244578    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244581    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244585    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244588    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244591    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244595    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244598    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244601    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244604    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244606    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244609    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244612    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244614    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244617    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244621    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244624    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244628    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 17:17:03.249288 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244633    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244638    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244641    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244644    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244648    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244650    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244653    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244656    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244659    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244662    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244665    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244667    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244670    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244673    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244675    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244678    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244681    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244683    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244686    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 17:17:03.249764 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244688    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244691    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244693    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.244696    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244766    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244796    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244801    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244806    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244811    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244814    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244818    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244822    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244825    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244828    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244833    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244836    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244839    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244842    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244845    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244848    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244850    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244853    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244858    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244861    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 17:17:03.250246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244864    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244867    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244871    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244879    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244882    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244884    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244888    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244891    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244894    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244897    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244900    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244902    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244906    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244909    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244912    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244915    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244918    2558 flags.go:64] FLAG: --enable-server="true"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244921    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244925    2558 flags.go:64] FLAG: --event-burst="100"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244928    2558 flags.go:64] FLAG: --event-qps="50"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244931    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244933    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244937    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244940    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 17:17:03.250823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244943    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244946    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244949    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244952    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244955    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244958    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244961    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244963    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244966    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244969    2558 flags.go:64] FLAG: --feature-gates=""
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244972    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244975    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244978    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244981    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.244984    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245000    2558 flags.go:64] FLAG: --help="false"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245003    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245006    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245009    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245019    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245023    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245027    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245030    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 17:17:03.251461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245032    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245035    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245038    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245042    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245045    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245067    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245071    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245074    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245077    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245079    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245084    2558 flags.go:64] FLAG: --lock-file=""
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245090    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245093    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245096    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245101    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245104    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245107    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245110    2558 flags.go:64] FLAG: --logging-format="text"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245113    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245116    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245119    2558 flags.go:64] FLAG: --manifest-url=""
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245122    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245127    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245129    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245134    2558 flags.go:64] FLAG: --max-pods="110"
Sep 09 17:17:03.252095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245136    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245140    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245142    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245145    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245148    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245151    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245155    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245162    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245165    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245168    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245171    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245175    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245180    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245183    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245186    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245189    2558 flags.go:64] FLAG: --port="10250"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245192    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245196    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-099209892d5752d5d"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245199    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245202    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245205    2558 flags.go:64] FLAG: --register-node="true"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245208    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245211    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245214    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 09 17:17:03.252695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245217    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245220    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245223    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245227    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245230    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245232    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245235    2558 flags.go:64] FLAG: --runonce="false"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245238    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245241    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245244    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245247    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245251    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245254    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245257    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245260    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245262    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245265    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245268    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245271    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245274    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245277    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245280    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245285    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245288    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245291    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 17:17:03.253301 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245296    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245299    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245302    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245304    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245307    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245310    2558 flags.go:64] FLAG: --v="2"
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245314    2558 flags.go:64] FLAG: --version="false"
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245318    2558 flags.go:64] FLAG: --vmodule=""
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245321    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.245324    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245441    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245444    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245448    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245451    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245453    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245456    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245458    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245461    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245464    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245466    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245469    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245471    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 17:17:03.253912 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245474    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245477    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245480    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245483    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245486    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245488    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245494    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245497    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245499    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245502    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245506    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245509    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245511    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245514    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245517    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245519    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245522    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245525    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245527    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245530    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 17:17:03.254464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245532    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245535    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245537    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245540    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245542    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245545    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245547    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245550    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245552    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245555    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245558    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245560    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245563    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245565    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245569    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245572    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245574    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245577    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245581    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 17:17:03.254985 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245585    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245588    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245591    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245595    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245597    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245600    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245602    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245605    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245607    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245610    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245612    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245615    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245618    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245620    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245623    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245625    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245628    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245630    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245633    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245636    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 17:17:03.255464 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245638    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245641    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245643    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245646    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245649    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245651    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245653    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245657    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245660    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245662    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245665    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245667    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245670    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245673    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245675    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245679    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245683    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245686    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245689    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 17:17:03.255943 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245692    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245695    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245698    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245702    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245705    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245707    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245710    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245713    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.245715    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.246324    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.253198    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.253215    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.253236    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.253242    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 17:17:03.256528 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253304    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253310    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253313    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253316    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253319    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253322    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253326    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253329    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253331    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253334    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253336    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253339    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253342    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253345    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253347    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253350    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253353    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253355    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253358    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253360    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253363    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 17:17:03.256904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253366    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253368    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253371    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253374    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253376    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253378    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253382    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253386    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253395    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253398    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253400    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253403    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253405    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253408    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253411    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253413    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253416    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253419    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253422    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 17:17:03.257444 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253424    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253427    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253429    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253432    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253434    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253437    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253439    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253442    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253445    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253448    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253450    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253453    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253456    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253458    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253461    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253463    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253466    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253468    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253471    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 17:17:03.257921 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253474    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253476    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253479    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253487    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253490    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253493    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253495    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253498    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253500    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253504    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253508    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253511    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253513    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253516    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253519    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253522    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253525    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253527    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253530    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 17:17:03.258414 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253533    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253535    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253538    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253542    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253545    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253548    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253550    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253553    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253555    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253558    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253560    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253563    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253566    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253569    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253571    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253574    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253577    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253584    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253587    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253590    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 17:17:03.258904 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253592    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.253597    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253739    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253744    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253746    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253749    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253752    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253755    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253758    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253760    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253763    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253766    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253769    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253771    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253774    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 17:17:03.259462 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253776    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253779    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253781    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253784    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253786    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253789    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253792    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253795    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253798    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253800    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253803    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253805    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253808    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253811    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253813    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253821    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253824    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253827    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253830    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253833    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 17:17:03.259821 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253835    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253838    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253840    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253843    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253847    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253850    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253853    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253856    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253858    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253861    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253864    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253866    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253869    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253871    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253874    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253876    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253878    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253881    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253884    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 17:17:03.260304 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253887    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253889    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253892    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253894    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253897    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253900    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253903    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253905    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253908    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253916    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253920    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253923    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253926    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253929    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253931    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253934    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253936    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253938    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253941    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 17:17:03.260767 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253943    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253946    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253948    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253951    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253953    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253956    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253960    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253963    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253966    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253969    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253971    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253974    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253977    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253980    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253982    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.253985    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254001    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254005    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254007    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 17:17:03.261260 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254010    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254013    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254015    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254018    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254027    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254030    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254033    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254035    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:03.254038    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.254043    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.254170    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.256055    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.256701    2558 server.go:1019] "Starting client certificate rotation"
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.256797    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 17:17:03.261709 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.257496    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 17:17:03.262076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.259512    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 17:17:03.262331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.262295    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 17:17:03.273229 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.273209    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 09 17:17:03.277239 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.277222    2558 log.go:25] "Validated CRI v1 image API"
Sep 09 17:17:03.278169 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.278152    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 17:17:03.283190 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.283165    2558 fs.go:135] Filesystem UUIDs: map[6da60508-dc73-4313-a1c4-92eb44559313:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 84c8c298-3bee-4ebf-814f-d100fc16176d:/dev/nvme0n1p3]
Sep 09 17:17:03.283259 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.283190    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 17:17:03.287458 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.287440    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 17:17:03.289038 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.288907    2558 manager.go:216] Machine: {Timestamp:2026-09-09 17:17:03.28782004 +0000 UTC m=+0.269936064 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103839 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d093ddde02665a5f22f6bac4eb86e SystemUUID:ec2d093d-dde0-2665-a5f2-2f6bac4eb86e BootID:f542d756-839d-4a7f-80f7-d29a388d0ae7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a8:fa:7a:ed:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a8:fa:7a:ed:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:ce:f2:0d:bb:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 17:17:03.289038 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.289034    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 17:17:03.289177 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.289167    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 17:17:03.290100 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.290072    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 17:17:03.290243 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.290102    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 17:17:03.290769 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.290760    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 17:17:03.290818 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.290770    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 17:17:03.290818 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.290779    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 17:17:03.291362 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.291352    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 17:17:03.292366 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.292356    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 17:17:03.295021 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295010    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 17:17:03.295133 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295123    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 17:17:03.295169 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295152    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 17:17:03.295169 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295162    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 17:17:03.295243 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295176    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 17:17:03.295243 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295185    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 09 17:17:03.295243 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295197    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 17:17:03.295519 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295503    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 17:17:03.295619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.295529    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 17:17:03.297418 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.297386    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 17:17:03.299142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299127    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 17:17:03.299213 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299157    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 17:17:03.299807 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299796    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 17:17:03.299842 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299813    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 17:17:03.299842 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299820    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 17:17:03.299842 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299825    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 17:17:03.299842 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299833    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 17:17:03.299842 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299838    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 17:17:03.299982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299851    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 17:17:03.299982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299856    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 17:17:03.299982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299863    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 17:17:03.299982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299868    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 17:17:03.299982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299877    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 17:17:03.299982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.299882    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 17:17:03.300597 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.300587    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 17:17:03.300597 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.300597    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 17:17:03.304615 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.304601    2558 server.go:1268] "Started kubelet"
Sep 09 17:17:03.304683 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.304661    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 17:17:03.308267 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.304683    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 17:17:03.308267 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.308116    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 17:17:03.309389 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.308756    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 17:17:03.309389 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.309339    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 17:17:03.308840 ip-10-0-139-100 systemd[1]: Started Kubernetes Kubelet.
Sep 09 17:17:03.309695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.309630    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 17:17:03.309983 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.309802    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 17:17:03.313133 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.313109    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjvq6"
Sep 09 17:17:03.313748 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.313729    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 17:17:03.315574 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.313616    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-100.ec2.internal.18d3b693e466371c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-100.ec2.internal,UID:ip-10-0-139-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-100.ec2.internal,},FirstTimestamp:2026-09-09 17:17:03.304570652 +0000 UTC m=+0.286686674,LastTimestamp:2026-09-09 17:17:03.304570652 +0000 UTC m=+0.286686674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-100.ec2.internal,}"
Sep 09 17:17:03.315574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.315563    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 17:17:03.316093 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.316075    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 17:17:03.316250 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.316217    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:03.316895 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.316874    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 17:17:03.317009 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.316904    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 17:17:03.317009 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.316925    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 17:17:03.317122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317051    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 17:17:03.317122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317060    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 17:17:03.317307 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317288    2558 factory.go:153] Registering CRI-O factory
Sep 09 17:17:03.317392 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317310    2558 factory.go:223] Registration of the crio container factory successfully
Sep 09 17:17:03.317392 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317367    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 17:17:03.317392 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317376    2558 factory.go:55] Registering systemd factory
Sep 09 17:17:03.317392 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317383    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 09 17:17:03.317582 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317398    2558 factory.go:103] Registering Raw factory
Sep 09 17:17:03.317582 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.317401    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 17:17:03.317582 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317415    2558 manager.go:1196] Started watching for new ooms in manager
Sep 09 17:17:03.317791 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.317774    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjvq6"
Sep 09 17:17:03.318347 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.318331    2558 manager.go:349] Starting recovery of all containers
Sep 09 17:17:03.328203 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.328186    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:03.328899 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.328884    2558 manager.go:354] Recovery completed
Sep 09 17:17:03.331157 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.331133    2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.333303 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.333288    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 17:17:03.335201 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.335187    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 17:17:03.335265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.335213    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 17:17:03.335265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.335224    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Sep 09 17:17:03.335799 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.335787    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 17:17:03.335867 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.335798    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 17:17:03.335867 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.335831    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 17:17:03.335867 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.335801    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 17:17:03.336573 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.336556    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 17:17:03.336631 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.336583    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 17:17:03.336631 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.336593    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Sep 09 17:17:03.336896 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.336886    2558 policy_none.go:49] "None policy: Start"
Sep 09 17:17:03.336950 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.336902    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 17:17:03.336950 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.336930    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 17:17:03.337847 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.337837    2558 policy_none.go:47] "Start"
Sep 09 17:17:03.369690 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.369673    2558 manager.go:337] "Starting Device Plugin manager"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.369715    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.369730    2558 server.go:87] "Starting device plugin registration server"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.369985    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.370012    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.370121    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.370204    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.370213    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.370857    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 17:17:03.376382 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.370911    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:03.454695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.454651    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 17:17:03.455895 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.455877    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 17:17:03.455986 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.455943    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 17:17:03.455986 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.455972    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 17:17:03.456107 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.456035    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 17:17:03.458340 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.458321    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:03.471097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.471082    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 17:17:03.471877 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.471859    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 17:17:03.471954 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.471890    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 17:17:03.471954 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.471904    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Sep 09 17:17:03.471954 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.471927    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.479558 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.479544    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.479625 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.479563    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-100.ec2.internal\": node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:03.498941 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.498850    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:03.557101 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.557066    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"]
Sep 09 17:17:03.557218 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.557141    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 17:17:03.558582 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.558566    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 17:17:03.558645 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.558595    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 17:17:03.558645 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.558604    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Sep 09 17:17:03.559612 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.559600    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 17:17:03.559746 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.559731    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.559789 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.559761    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 17:17:03.560321 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.560304    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 17:17:03.560417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.560335    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 17:17:03.560417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.560304    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 17:17:03.560417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.560347    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Sep 09 17:17:03.560417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.560384    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 17:17:03.560417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.560400    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Sep 09 17:17:03.561569 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.561555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.561642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.561579    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 17:17:03.562245 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.562216    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 17:17:03.562364 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.562256    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 17:17:03.562364 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.562269    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeHasSufficientPID"
Sep 09 17:17:03.587522 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.587503    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.591790 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.591775    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-100.ec2.internal\" not found" node="ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.599087 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.599074    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:03.699362 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.699323    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:03.718630 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.718604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad0f471830ea8bb8eada66ef51ef7e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9ad0f471830ea8bb8eada66ef51ef7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.718691 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.718638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad0f471830ea8bb8eada66ef51ef7e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9ad0f471830ea8bb8eada66ef51ef7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.718691 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.718654    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a63bc5640b9cbe7de0e68e356f93dd93-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a63bc5640b9cbe7de0e68e356f93dd93\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.800049 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.799967    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:03.819296 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.819269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad0f471830ea8bb8eada66ef51ef7e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9ad0f471830ea8bb8eada66ef51ef7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.819358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.819300    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad0f471830ea8bb8eada66ef51ef7e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9ad0f471830ea8bb8eada66ef51ef7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.819358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.819316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a63bc5640b9cbe7de0e68e356f93dd93-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a63bc5640b9cbe7de0e68e356f93dd93\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.819419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.819378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ad0f471830ea8bb8eada66ef51ef7e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9ad0f471830ea8bb8eada66ef51ef7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.819419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.819393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a63bc5640b9cbe7de0e68e356f93dd93-config\") pod \"kube-apiserver-proxy-ip-10-0-139-100.ec2.internal\" (UID: \"a63bc5640b9cbe7de0e68e356f93dd93\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.819481 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.819378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ad0f471830ea8bb8eada66ef51ef7e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal\" (UID: \"9ad0f471830ea8bb8eada66ef51ef7e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.889469 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.889435    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.894274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:03.894255    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:03.901117 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:03.901092    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:04.001618 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:04.001579    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:05.456105 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.456081    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:05.527882 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.527850    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:05.556800 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.556773    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-100.ec2.internal\" not found"
Sep 09 17:17:05.587319 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.587293    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:05.611001 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.610967    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 17:17:05.611391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.611094    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 17:17:05.611391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.611124    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 17:17:05.611391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.611150    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 17:17:05.611391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.611153    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 17:17:05.650282 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.650259    2558 apiserver.go:52] "Watching apiserver"
Sep 09 17:17:05.668532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.668508    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 17:17:05.670751 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.670719    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 17:17:05.670842 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.670816    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-tv95p","openshift-ovn-kubernetes/ovnkube-node-snbz8","kube-system/konnectivity-agent-qngb4","openshift-dns/node-resolver-pdwrx","openshift-image-registry/node-ca-qpr8j","openshift-multus/multus-c84jg","openshift-multus/network-metrics-daemon-zmm4h","openshift-network-operator/iptables-alerter-65xwz","kube-system/global-pull-secret-syncer-l8zcg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q","openshift-cluster-node-tuning-operator/tuned-rqqmz","openshift-multus/multus-additional-cni-plugins-ptt54"]
Sep 09 17:17:05.670920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.670735    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:05.672746 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.672725    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:05.673245 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.673220    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:05.673322 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.673278    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 17:12:04 +0000 UTC" deadline="2028-05-14 16:32:24.696911684 +0000 UTC"
Sep 09 17:17:05.673322 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.673314    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14711h15m19.02360006s"
Sep 09 17:17:05.674601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.674455    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.675779 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.675762    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:05.677256 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.676729    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:05.677503 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.677479    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.677649 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.677623    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 17:17:05.677712 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.677664    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 17:17:05.677764 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.677711    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gk86x\""
Sep 09 17:17:05.677764 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.677714    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 17:17:05.677859 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.677768    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 17:17:05.678188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.678169    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 17:17:05.678254 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.678199    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.678985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.678962    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rrsh8\""
Sep 09 17:17:05.679125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.679002    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 17:17:05.679630 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.679612    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 17:17:05.679713 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.679636    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 17:17:05.679909 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.679893    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:05.680031 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.679975    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:05.680242 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.680227    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 17:17:05.680332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.680242    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 17:17:05.680332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.680269    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 17:17:05.680419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.680376    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwkkc\""
Sep 09 17:17:05.680561 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.680548    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 17:17:05.681027 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.681015    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 17:17:05.681429 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.681412    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 17:17:05.681513 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.681491    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"
Sep 09 17:17:05.681574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.681553    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.681708 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.681577    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 17:17:05.681771 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.681732    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vlbqv\""
Sep 09 17:17:05.683022 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.683006    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.683124 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.683082    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:05.683871 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.683856    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 17:17:05.684080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.684064    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 17:17:05.684080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.684067    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 17:17:05.684252 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.684110    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q8xdw\""
Sep 09 17:17:05.684308 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.684300    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 17:17:05.684990 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.684947    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.686812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.686769    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.687412 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.687389    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 17:17:05.687641 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.687625    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 17:17:05.687694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.687665    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 17:17:05.687694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.687682    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xqwr7\""
Sep 09 17:17:05.688201 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.688182    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.688939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.688923    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 17:17:05.689424 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.689396    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 17:17:05.689424 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.689411    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hb4b5\""
Sep 09 17:17:05.690208 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.690188    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal"]
Sep 09 17:17:05.690291 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.690279    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.690538 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.690510    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 17:17:05.690742 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.690728    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 17:17:05.690824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.690772    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mwv8w\""
Sep 09 17:17:05.692044 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.692024    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 17:17:05.692157 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.692141    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal"]
Sep 09 17:17:05.693931 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.693262    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 17:17:05.693931 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.693347    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 17:17:05.693931 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.693412    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ptp86\""
Sep 09 17:17:05.693931 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.693826    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 17:17:05.702423 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.702404    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9hsx4"
Sep 09 17:17:05.709792 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.709776    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9hsx4"
Sep 09 17:17:05.771698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.771674    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 17:17:05.776998 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.776969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-cni-netd\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b07f3322-92ad-4575-af6a-ae77de513515-serviceca\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.777122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysctl-d\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.777122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777083    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fft7\" (UniqueName: \"kubernetes.io/projected/92691a45-2af1-48ac-830a-ead94a5117e6-kube-api-access-6fft7\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:05.777314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.777314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2896185f-281a-482f-a8ad-7a4895391831-host-slash\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.777314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777215    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-modprobe-d\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.777314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-node-log\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-log-socket\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-netns\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.777314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777293    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-cni-bin\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777318    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-slash\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-run-netns\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9sb\" (UniqueName: \"kubernetes.io/projected/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-kube-api-access-6m9sb\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-socket-dir-parent\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07f3322-92ad-4575-af6a-ae77de513515-host\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/849b01c7-caef-49a5-b7e2-b2274e4ff53b-hosts-file\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777494    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c38f48-b836-41fb-92a1-2190725d2d18-kubelet-config\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777516    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c38f48-b836-41fb-92a1-2190725d2d18-dbus\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-var-lib-kubelet\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.777551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a374f5-cf67-4db9-bd80-518a808fcd26-agent-certs\") pod \"konnectivity-agent-qngb4\" (UID: \"19a374f5-cf67-4db9-bd80-518a808fcd26\") " pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777590    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-env-overrides\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777611    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-run\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovnkube-script-lib\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777662    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782p7\" (UniqueName: \"kubernetes.io/projected/790c0016-2adc-467b-9a36-9def50ac5db4-kube-api-access-782p7\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-tuned\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-etc-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-system-cni-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777723    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-hostroot\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777748    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-socket-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777780    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-tmp\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777807    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-ovn\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-run-ovn-kubernetes\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-daemon-config\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777870    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-etc-selinux\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.777948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777901    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-system-cni-dir\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777919    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zh5\" (UniqueName: \"kubernetes.io/projected/849b01c7-caef-49a5-b7e2-b2274e4ff53b-kube-api-access-29zh5\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.777998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-systemd\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovn-node-metrics-cert\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-cnibin\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-device-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778127    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-sys-fs\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b01c7-caef-49a5-b7e2-b2274e4ff53b-tmp-dir\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778176    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-lib-modules\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778208    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-host\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778223    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-kubelet\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovnkube-config\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-k8s-cni-cncf-io\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-cni-multus\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778278    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-etc-kubernetes\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.778547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778312    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2wg\" (UniqueName: \"kubernetes.io/projected/5a428b72-d41b-4a99-a171-75bfccbbe1b9-kube-api-access-wp2wg\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6zbw\" (UniqueName: \"kubernetes.io/projected/cf0778c3-c4ce-42d6-b7e0-51544b593949-kube-api-access-g6zbw\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778402    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-kubelet\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778427    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-registration-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778453    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gknv\" (UniqueName: \"kubernetes.io/projected/b07f3322-92ad-4575-af6a-ae77de513515-kube-api-access-6gknv\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2896185f-281a-482f-a8ad-7a4895391831-iptables-alerter-script\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtm89\" (UniqueName: \"kubernetes.io/projected/2896185f-281a-482f-a8ad-7a4895391831-kube-api-access-qtm89\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778526    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysconfig\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778557    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-systemd\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-systemd-units\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778610    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-var-lib-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-cni-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-multus-certs\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cht\" (UniqueName: \"kubernetes.io/projected/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-kube-api-access-d9cht\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778711    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-kubernetes\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.779148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778733    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysctl-conf\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778763    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-sys\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778797    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a428b72-d41b-4a99-a171-75bfccbbe1b9-cni-binary-copy\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-conf-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-cnibin\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778930    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a374f5-cf67-4db9-bd80-518a808fcd26-konnectivity-ca\") pod \"konnectivity-agent-qngb4\" (UID: \"19a374f5-cf67-4db9-bd80-518a808fcd26\") " pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.778996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:05.779752 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.779052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-cni-bin\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.784601 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.784585    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 17:17:05.784682 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.784606    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 17:17:05.784682 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.784616    2558 projected.go:196] Error preparing data for projected volume kube-api-access-86gqv for pod openshift-network-diagnostics/network-check-target-tv95p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:05.784682 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.784669    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv podName:042adbf6-fefd-465e-a52c-373c0491c7ba nodeName:}" failed. No retries permitted until 2026-09-09 17:17:06.284652447 +0000 UTC m=+1.912805149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-86gqv" (UniqueName: "kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv") pod "network-check-target-tv95p" (UID: "042adbf6-fefd-465e-a52c-373c0491c7ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:05.817147 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:05.817099    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63bc5640b9cbe7de0e68e356f93dd93.slice/crio-501992f71d3153e2c61ede1d442ca25cb18a871c85e16989ee82a1267abc2fce WatchSource:0}: Error finding container 501992f71d3153e2c61ede1d442ca25cb18a871c85e16989ee82a1267abc2fce: Status 404 returned error can't find the container with id 501992f71d3153e2c61ede1d442ca25cb18a871c85e16989ee82a1267abc2fce
Sep 09 17:17:05.817334 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:05.817316    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad0f471830ea8bb8eada66ef51ef7e5.slice/crio-453e9f9ee3dcaa6c94ae11e8379d1d837792f7de3aeeb47a39c4f3cdbdd3aadb WatchSource:0}: Error finding container 453e9f9ee3dcaa6c94ae11e8379d1d837792f7de3aeeb47a39c4f3cdbdd3aadb: Status 404 returned error can't find the container with id 453e9f9ee3dcaa6c94ae11e8379d1d837792f7de3aeeb47a39c4f3cdbdd3aadb
Sep 09 17:17:05.823509 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.822818    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 17:17:05.880288 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b01c7-caef-49a5-b7e2-b2274e4ff53b-tmp-dir\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.880288 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-lib-modules\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-host\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-kubelet\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovnkube-config\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-k8s-cni-cncf-io\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-host\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-kubelet\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-cni-multus\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880440    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-lib-modules\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880449    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-cni-multus\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-etc-kubernetes\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880458    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-k8s-cni-cncf-io\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2wg\" (UniqueName: \"kubernetes.io/projected/5a428b72-d41b-4a99-a171-75bfccbbe1b9-kube-api-access-wp2wg\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.880516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6zbw\" (UniqueName: \"kubernetes.io/projected/cf0778c3-c4ce-42d6-b7e0-51544b593949-kube-api-access-g6zbw\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-etc-kubernetes\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-kubelet\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-registration-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880591    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gknv\" (UniqueName: \"kubernetes.io/projected/b07f3322-92ad-4575-af6a-ae77de513515-kube-api-access-6gknv\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-kubelet\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2896185f-281a-482f-a8ad-7a4895391831-iptables-alerter-script\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtm89\" (UniqueName: \"kubernetes.io/projected/2896185f-281a-482f-a8ad-7a4895391831-kube-api-access-qtm89\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-registration-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysconfig\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysconfig\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880753    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-systemd\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880797    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-systemd\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-systemd-units\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880823    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-var-lib-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.881125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-systemd-units\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-cni-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-multus-certs\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880875    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cht\" (UniqueName: \"kubernetes.io/projected/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-kube-api-access-d9cht\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-var-lib-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-kubernetes\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysctl-conf\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-multus-certs\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.880993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-sys\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a428b72-d41b-4a99-a171-75bfccbbe1b9-cni-binary-copy\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881033    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-kubernetes\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-conf-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-cni-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysctl-conf\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-sys\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-conf-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881158    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2896185f-281a-482f-a8ad-7a4895391831-iptables-alerter-script\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:05.881928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881187    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-cnibin\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a374f5-cf67-4db9-bd80-518a808fcd26-konnectivity-ca\") pod \"konnectivity-agent-qngb4\" (UID: \"19a374f5-cf67-4db9-bd80-518a808fcd26\") " pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881240    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-cnibin\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.881247    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-cni-bin\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-cni-netd\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b07f3322-92ad-4575-af6a-ae77de513515-serviceca\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-cni-bin\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysctl-d\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881391    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b01c7-caef-49a5-b7e2-b2274e4ff53b-tmp-dir\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881430    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-cni-netd\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.881397    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:06.381376436 +0000 UTC m=+2.009529127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-sysctl-d\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fft7\" (UniqueName: \"kubernetes.io/projected/92691a45-2af1-48ac-830a-ead94a5117e6-kube-api-access-6fft7\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:05.882741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2896185f-281a-482f-a8ad-7a4895391831-host-slash\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881594    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-modprobe-d\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2896185f-281a-482f-a8ad-7a4895391831-host-slash\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-node-log\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-log-socket\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881646    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a428b72-d41b-4a99-a171-75bfccbbe1b9-cni-binary-copy\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-netns\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-cni-bin\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-node-log\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881707    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovnkube-config\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-slash\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881742    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a374f5-cf67-4db9-bd80-518a808fcd26-konnectivity-ca\") pod \"konnectivity-agent-qngb4\" (UID: \"19a374f5-cf67-4db9-bd80-518a808fcd26\") " pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881780    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-run-netns\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-run-netns\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881783    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-modprobe-d\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9sb\" (UniqueName: \"kubernetes.io/projected/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-kube-api-access-6m9sb\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.883438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-host-var-lib-cni-bin\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881868    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b07f3322-92ad-4575-af6a-ae77de513515-serviceca\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881878    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-run-netns\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-log-socket\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-socket-dir-parent\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-socket-dir-parent\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881976    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07f3322-92ad-4575-af6a-ae77de513515-host\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.881999    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-slash\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/849b01c7-caef-49a5-b7e2-b2274e4ff53b-hosts-file\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882039    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/849b01c7-caef-49a5-b7e2-b2274e4ff53b-hosts-file\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882046    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c38f48-b836-41fb-92a1-2190725d2d18-kubelet-config\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07f3322-92ad-4575-af6a-ae77de513515-host\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.882062    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882073    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c38f48-b836-41fb-92a1-2190725d2d18-dbus\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882105    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c38f48-b836-41fb-92a1-2190725d2d18-kubelet-config\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.884200 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:05.882119    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret podName:56c38f48-b836-41fb-92a1-2190725d2d18 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:06.382108136 +0000 UTC m=+2.010260817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret") pod "global-pull-secret-syncer-l8zcg" (UID: "56c38f48-b836-41fb-92a1-2190725d2d18") : object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-var-lib-kubelet\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882150    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c38f48-b836-41fb-92a1-2190725d2d18-dbus\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a374f5-cf67-4db9-bd80-518a808fcd26-agent-certs\") pod \"konnectivity-agent-qngb4\" (UID: \"19a374f5-cf67-4db9-bd80-518a808fcd26\") " pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-env-overrides\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-var-lib-kubelet\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-run\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovnkube-script-lib\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-run\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782p7\" (UniqueName: \"kubernetes.io/projected/790c0016-2adc-467b-9a36-9def50ac5db4-kube-api-access-782p7\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf0778c3-c4ce-42d6-b7e0-51544b593949-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-tuned\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-etc-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-system-cni-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.885076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-hostroot\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-socket-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882447    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-etc-openvswitch\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-tmp\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-ovn\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-env-overrides\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-run-ovn-kubernetes\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-host-run-ovn-kubernetes\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-daemon-config\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-ovn\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-etc-selinux\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-system-cni-dir\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882629    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-socket-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882631    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-hostroot\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-system-cni-dir\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882689    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882707    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-etc-selinux\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.885845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovnkube-script-lib\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zh5\" (UniqueName: \"kubernetes.io/projected/849b01c7-caef-49a5-b7e2-b2274e4ff53b-kube-api-access-29zh5\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-systemd\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-system-cni-dir\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovn-node-metrics-cert\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882726    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-cnibin\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-device-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-sys-fs\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf0778c3-c4ce-42d6-b7e0-51544b593949-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882795    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-run-systemd\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-sys-fs\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a428b72-d41b-4a99-a171-75bfccbbe1b9-cnibin\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.882993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/790c0016-2adc-467b-9a36-9def50ac5db4-device-dir\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.883060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a428b72-d41b-4a99-a171-75bfccbbe1b9-multus-daemon-config\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.886061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-tmp\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.886079    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-etc-tuned\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.886217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-ovn-node-metrics-cert\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.886407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.886325    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a374f5-cf67-4db9-bd80-518a808fcd26-agent-certs\") pod \"konnectivity-agent-qngb4\" (UID: \"19a374f5-cf67-4db9-bd80-518a808fcd26\") " pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:05.888741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.888717    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6zbw\" (UniqueName: \"kubernetes.io/projected/cf0778c3-c4ce-42d6-b7e0-51544b593949-kube-api-access-g6zbw\") pod \"multus-additional-cni-plugins-ptt54\" (UID: \"cf0778c3-c4ce-42d6-b7e0-51544b593949\") " pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:05.890549 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.890522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gknv\" (UniqueName: \"kubernetes.io/projected/b07f3322-92ad-4575-af6a-ae77de513515-kube-api-access-6gknv\") pod \"node-ca-qpr8j\" (UID: \"b07f3322-92ad-4575-af6a-ae77de513515\") " pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:05.891142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.891119    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtm89\" (UniqueName: \"kubernetes.io/projected/2896185f-281a-482f-a8ad-7a4895391831-kube-api-access-qtm89\") pod \"iptables-alerter-65xwz\" (UID: \"2896185f-281a-482f-a8ad-7a4895391831\") " pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:05.891246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.891178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cht\" (UniqueName: \"kubernetes.io/projected/238714e6-85d5-4af3-9cb8-d4f87ef2b4de-kube-api-access-d9cht\") pod \"tuned-rqqmz\" (UID: \"238714e6-85d5-4af3-9cb8-d4f87ef2b4de\") " pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:05.891358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.891328    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fft7\" (UniqueName: \"kubernetes.io/projected/92691a45-2af1-48ac-830a-ead94a5117e6-kube-api-access-6fft7\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:05.891461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.891369    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2wg\" (UniqueName: \"kubernetes.io/projected/5a428b72-d41b-4a99-a171-75bfccbbe1b9-kube-api-access-wp2wg\") pod \"multus-c84jg\" (UID: \"5a428b72-d41b-4a99-a171-75bfccbbe1b9\") " pod="openshift-multus/multus-c84jg"
Sep 09 17:17:05.891525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.891491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-782p7\" (UniqueName: \"kubernetes.io/projected/790c0016-2adc-467b-9a36-9def50ac5db4-kube-api-access-782p7\") pod \"aws-ebs-csi-driver-node-fg48q\" (UID: \"790c0016-2adc-467b-9a36-9def50ac5db4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:05.892195 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.892177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9sb\" (UniqueName: \"kubernetes.io/projected/589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8-kube-api-access-6m9sb\") pod \"ovnkube-node-snbz8\" (UID: \"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:05.892459 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:05.892441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zh5\" (UniqueName: \"kubernetes.io/projected/849b01c7-caef-49a5-b7e2-b2274e4ff53b-kube-api-access-29zh5\") pod \"node-resolver-pdwrx\" (UID: \"849b01c7-caef-49a5-b7e2-b2274e4ff53b\") " pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:06.002207 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.002106    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:06.010666 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.010643    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589eb0f8_9a41_4c82_bb43_e7b0e1e4a2b8.slice/crio-0b2b7cfa5f5e9bd91d712133fb60cc48ff44718471277798595e28015997c51f WatchSource:0}: Error finding container 0b2b7cfa5f5e9bd91d712133fb60cc48ff44718471277798595e28015997c51f: Status 404 returned error can't find the container with id 0b2b7cfa5f5e9bd91d712133fb60cc48ff44718471277798595e28015997c51f
Sep 09 17:17:06.017834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.017812    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:06.024776 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.024756    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a374f5_cf67_4db9_bd80_518a808fcd26.slice/crio-2c2b481ddf80baedbfd608536f9ac95bfee2cf080ceeec3b80b69ec1ab85df2d WatchSource:0}: Error finding container 2c2b481ddf80baedbfd608536f9ac95bfee2cf080ceeec3b80b69ec1ab85df2d: Status 404 returned error can't find the container with id 2c2b481ddf80baedbfd608536f9ac95bfee2cf080ceeec3b80b69ec1ab85df2d
Sep 09 17:17:06.032658 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.032643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pdwrx"
Sep 09 17:17:06.039030 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.038989    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849b01c7_caef_49a5_b7e2_b2274e4ff53b.slice/crio-50f802c9e57f28a2d9dddfff513bc16fb3a2bdc4e2c40f797a5cad5d7e784eeb WatchSource:0}: Error finding container 50f802c9e57f28a2d9dddfff513bc16fb3a2bdc4e2c40f797a5cad5d7e784eeb: Status 404 returned error can't find the container with id 50f802c9e57f28a2d9dddfff513bc16fb3a2bdc4e2c40f797a5cad5d7e784eeb
Sep 09 17:17:06.048980 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.048949    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c84jg"
Sep 09 17:17:06.055324 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.055303    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a428b72_d41b_4a99_a171_75bfccbbe1b9.slice/crio-58f0a4a91f5ef67244abe28aba28684e1cd34a7d76c4a010b29598119389cd87 WatchSource:0}: Error finding container 58f0a4a91f5ef67244abe28aba28684e1cd34a7d76c4a010b29598119389cd87: Status 404 returned error can't find the container with id 58f0a4a91f5ef67244abe28aba28684e1cd34a7d76c4a010b29598119389cd87
Sep 09 17:17:06.061583 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.061566    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65xwz"
Sep 09 17:17:06.068103 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.068082    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2896185f_281a_482f_a8ad_7a4895391831.slice/crio-c38dc542aa1cd0da619425ae5977adae232cb732392100d924a58d1887967154 WatchSource:0}: Error finding container c38dc542aa1cd0da619425ae5977adae232cb732392100d924a58d1887967154: Status 404 returned error can't find the container with id c38dc542aa1cd0da619425ae5977adae232cb732392100d924a58d1887967154
Sep 09 17:17:06.079355 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.079339    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q"
Sep 09 17:17:06.085716 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.085696    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790c0016_2adc_467b_9a36_9def50ac5db4.slice/crio-35f82f5f121f518b743c9a1b437dc1c09b19f17af7dbdeaa4468998d7ca499c5 WatchSource:0}: Error finding container 35f82f5f121f518b743c9a1b437dc1c09b19f17af7dbdeaa4468998d7ca499c5: Status 404 returned error can't find the container with id 35f82f5f121f518b743c9a1b437dc1c09b19f17af7dbdeaa4468998d7ca499c5
Sep 09 17:17:06.098009 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.097948    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rqqmz"
Sep 09 17:17:06.104545 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.104524    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238714e6_85d5_4af3_9cb8_d4f87ef2b4de.slice/crio-3a1a8247aa5486937ce5ab0ee8aff5dbc38cb0029941345ec6ba5b24131be40c WatchSource:0}: Error finding container 3a1a8247aa5486937ce5ab0ee8aff5dbc38cb0029941345ec6ba5b24131be40c: Status 404 returned error can't find the container with id 3a1a8247aa5486937ce5ab0ee8aff5dbc38cb0029941345ec6ba5b24131be40c
Sep 09 17:17:06.120851 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.120817    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptt54"
Sep 09 17:17:06.128143 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.128121    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0778c3_c4ce_42d6_b7e0_51544b593949.slice/crio-47d0e75c9867e7e313314696df83fccecef5933351bb3f2ba3884d1ba62c60c9 WatchSource:0}: Error finding container 47d0e75c9867e7e313314696df83fccecef5933351bb3f2ba3884d1ba62c60c9: Status 404 returned error can't find the container with id 47d0e75c9867e7e313314696df83fccecef5933351bb3f2ba3884d1ba62c60c9
Sep 09 17:17:06.128651 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.128624    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qpr8j"
Sep 09 17:17:06.135688 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:06.135662    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07f3322_92ad_4575_af6a_ae77de513515.slice/crio-838bda72b54e31f61ba21fe264d4206bd9ed6d5c23651904ce5ad2e6f4e4d0f4 WatchSource:0}: Error finding container 838bda72b54e31f61ba21fe264d4206bd9ed6d5c23651904ce5ad2e6f4e4d0f4: Status 404 returned error can't find the container with id 838bda72b54e31f61ba21fe264d4206bd9ed6d5c23651904ce5ad2e6f4e4d0f4
Sep 09 17:17:06.286248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.286177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:06.286386 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.286298    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 17:17:06.286386 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.286314    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 17:17:06.286386 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.286323    2558 projected.go:196] Error preparing data for projected volume kube-api-access-86gqv for pod openshift-network-diagnostics/network-check-target-tv95p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:06.286487 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.286392    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv podName:042adbf6-fefd-465e-a52c-373c0491c7ba nodeName:}" failed. No retries permitted until 2026-09-09 17:17:07.28637906 +0000 UTC m=+2.914531741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-86gqv" (UniqueName: "kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv") pod "network-check-target-tv95p" (UID: "042adbf6-fefd-465e-a52c-373c0491c7ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:06.387237 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.387207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:06.387367 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.387257    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:06.387416 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.387362    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:06.387446 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.387420    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:07.387406823 +0000 UTC m=+3.015559507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:06.387494 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.387362    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:06.387526 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:06.387497    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret podName:56c38f48-b836-41fb-92a1-2190725d2d18 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:07.387485992 +0000 UTC m=+3.015638677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret") pod "global-pull-secret-syncer-l8zcg" (UID: "56c38f48-b836-41fb-92a1-2190725d2d18") : object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:06.539106 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.539008    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:06.617586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.617552    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:06.712678 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.712453    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 17:12:05 +0000 UTC" deadline="2028-03-09 13:23:52.42456858 +0000 UTC"
Sep 09 17:17:06.712678 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.712499    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13124h6m45.71207606s"
Sep 09 17:17:06.845925 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.845790    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qpr8j" event={"ID":"b07f3322-92ad-4575-af6a-ae77de513515","Type":"ContainerStarted","Data":"838bda72b54e31f61ba21fe264d4206bd9ed6d5c23651904ce5ad2e6f4e4d0f4"}
Sep 09 17:17:06.871172 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.869449    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c84jg" event={"ID":"5a428b72-d41b-4a99-a171-75bfccbbe1b9","Type":"ContainerStarted","Data":"58f0a4a91f5ef67244abe28aba28684e1cd34a7d76c4a010b29598119389cd87"}
Sep 09 17:17:06.887570 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.887436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"0b2b7cfa5f5e9bd91d712133fb60cc48ff44718471277798595e28015997c51f"}
Sep 09 17:17:06.906379 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.906268    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerStarted","Data":"47d0e75c9867e7e313314696df83fccecef5933351bb3f2ba3884d1ba62c60c9"}
Sep 09 17:17:06.923807 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.923769    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rqqmz" event={"ID":"238714e6-85d5-4af3-9cb8-d4f87ef2b4de","Type":"ContainerStarted","Data":"3a1a8247aa5486937ce5ab0ee8aff5dbc38cb0029941345ec6ba5b24131be40c"}
Sep 09 17:17:06.932715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.932680    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q" event={"ID":"790c0016-2adc-467b-9a36-9def50ac5db4","Type":"ContainerStarted","Data":"35f82f5f121f518b743c9a1b437dc1c09b19f17af7dbdeaa4468998d7ca499c5"}
Sep 09 17:17:06.937788 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.937754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65xwz" event={"ID":"2896185f-281a-482f-a8ad-7a4895391831","Type":"ContainerStarted","Data":"c38dc542aa1cd0da619425ae5977adae232cb732392100d924a58d1887967154"}
Sep 09 17:17:06.950900 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.950868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pdwrx" event={"ID":"849b01c7-caef-49a5-b7e2-b2274e4ff53b","Type":"ContainerStarted","Data":"50f802c9e57f28a2d9dddfff513bc16fb3a2bdc4e2c40f797a5cad5d7e784eeb"}
Sep 09 17:17:06.954769 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.954729    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qngb4" event={"ID":"19a374f5-cf67-4db9-bd80-518a808fcd26","Type":"ContainerStarted","Data":"2c2b481ddf80baedbfd608536f9ac95bfee2cf080ceeec3b80b69ec1ab85df2d"}
Sep 09 17:17:06.966705 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.966662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"a63bc5640b9cbe7de0e68e356f93dd93","Type":"ContainerStarted","Data":"501992f71d3153e2c61ede1d442ca25cb18a871c85e16989ee82a1267abc2fce"}
Sep 09 17:17:06.976766 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:06.976719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"9ad0f471830ea8bb8eada66ef51ef7e5","Type":"ContainerStarted","Data":"453e9f9ee3dcaa6c94ae11e8379d1d837792f7de3aeeb47a39c4f3cdbdd3aadb"}
Sep 09 17:17:07.296180 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.296085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:07.296355 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.296295    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 17:17:07.296355 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.296316    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 17:17:07.296355 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.296329    2558 projected.go:196] Error preparing data for projected volume kube-api-access-86gqv for pod openshift-network-diagnostics/network-check-target-tv95p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:07.296509 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.296385    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv podName:042adbf6-fefd-465e-a52c-373c0491c7ba nodeName:}" failed. No retries permitted until 2026-09-09 17:17:09.296367566 +0000 UTC m=+4.924520248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-86gqv" (UniqueName: "kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv") pod "network-check-target-tv95p" (UID: "042adbf6-fefd-465e-a52c-373c0491c7ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:07.397455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.396724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:07.397455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.396795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:07.397455 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.396939    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:07.397455 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.397021    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret podName:56c38f48-b836-41fb-92a1-2190725d2d18 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:09.397002863 +0000 UTC m=+5.025155547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret") pod "global-pull-secret-syncer-l8zcg" (UID: "56c38f48-b836-41fb-92a1-2190725d2d18") : object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:07.397455 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.397105    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:07.397455 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.397141    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:09.397129514 +0000 UTC m=+5.025282198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:07.716205 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.716095    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 17:12:05 +0000 UTC" deadline="2028-02-26 22:49:48.661787741 +0000 UTC"
Sep 09 17:17:07.716205 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.716159    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12845h32m40.945633397s"
Sep 09 17:17:07.791142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.790889    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:07.811004 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.810967    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:07.811163 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.811116    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:07.811234 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.811203    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:07.811304 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.811286    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:07.811358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:07.811350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:07.811449 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:07.811424    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:08.696464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:08.696425    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 17:17:09.312245 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:09.312195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:09.312735 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.312486    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 17:17:09.312735 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.312506    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 17:17:09.312735 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.312518    2558 projected.go:196] Error preparing data for projected volume kube-api-access-86gqv for pod openshift-network-diagnostics/network-check-target-tv95p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:09.312735 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.312576    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv podName:042adbf6-fefd-465e-a52c-373c0491c7ba nodeName:}" failed. No retries permitted until 2026-09-09 17:17:13.3125581 +0000 UTC m=+8.940710784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-86gqv" (UniqueName: "kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv") pod "network-check-target-tv95p" (UID: "042adbf6-fefd-465e-a52c-373c0491c7ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:09.413426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:09.413369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:09.413597 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:09.413457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:09.413597 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.413586    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:09.413706 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.413648    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:13.413629338 +0000 UTC m=+9.041782018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:09.414067 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.414043    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:09.414167 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.414099    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret podName:56c38f48-b836-41fb-92a1-2190725d2d18 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:13.414084748 +0000 UTC m=+9.042237433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret") pod "global-pull-secret-syncer-l8zcg" (UID: "56c38f48-b836-41fb-92a1-2190725d2d18") : object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:09.810330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:09.810286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:09.810492 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.810415    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:09.810619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:09.810596    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:09.810751 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:09.810621    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:09.810751 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.810696    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:09.810870 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:09.810789    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:11.810472 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:11.810436    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:11.810907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:11.810488    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:11.810907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:11.810488    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:11.810907 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:11.810577    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:11.810907 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:11.810678    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:11.810907 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:11.810751    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:13.344500 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:13.344462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:13.344920 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.344650    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 17:17:13.344920 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.344676    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 17:17:13.344920 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.344688    2558 projected.go:196] Error preparing data for projected volume kube-api-access-86gqv for pod openshift-network-diagnostics/network-check-target-tv95p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:13.344920 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.344750    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv podName:042adbf6-fefd-465e-a52c-373c0491c7ba nodeName:}" failed. No retries permitted until 2026-09-09 17:17:21.344732149 +0000 UTC m=+16.972884834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-86gqv" (UniqueName: "kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv") pod "network-check-target-tv95p" (UID: "042adbf6-fefd-465e-a52c-373c0491c7ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:13.444864 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:13.444817    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:13.445066 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:13.444903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:13.445066 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.444940    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:13.445066 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.445019    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret podName:56c38f48-b836-41fb-92a1-2190725d2d18 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:21.444998064 +0000 UTC m=+17.073150748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret") pod "global-pull-secret-syncer-l8zcg" (UID: "56c38f48-b836-41fb-92a1-2190725d2d18") : object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:13.445066 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.445050    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:13.445274 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.445101    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:21.44508519 +0000 UTC m=+17.073237875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:13.810636 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:13.810600    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:13.810824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:13.810741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:13.810890 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.810847    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:13.810890 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.810737    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:13.811096 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:13.810893    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:13.811096 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:13.810991    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:15.810511 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:15.810475    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:15.811134 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:15.810470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:15.811134 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:15.810603    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:15.811134 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:15.810489    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:15.811134 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:15.810654    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:15.811134 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:15.810712    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:17.810477 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:17.810444    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:17.810941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:17.810492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:17.810941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:17.810492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:17.810941 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:17.810593    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:17.810941 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:17.810662    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:17.810941 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:17.810725    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:19.810451 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:19.810410    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:19.810906 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:19.810534    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:19.810906 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:19.810542    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:19.810906 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:19.810574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:19.810906 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:19.810668    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:19.810906 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:19.810724    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:21.403104 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:21.403062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:21.403584 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.403230    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 17:17:21.403584 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.403250    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 17:17:21.403584 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.403263    2558 projected.go:196] Error preparing data for projected volume kube-api-access-86gqv for pod openshift-network-diagnostics/network-check-target-tv95p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:21.403584 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.403324    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv podName:042adbf6-fefd-465e-a52c-373c0491c7ba nodeName:}" failed. No retries permitted until 2026-09-09 17:17:37.403307409 +0000 UTC m=+33.031460111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-86gqv" (UniqueName: "kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv") pod "network-check-target-tv95p" (UID: "042adbf6-fefd-465e-a52c-373c0491c7ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:21.504217 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:21.504179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:21.504380 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:21.504260    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:21.504380 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.504336    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:21.504380 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.504363    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:21.504534 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.504400    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret podName:56c38f48-b836-41fb-92a1-2190725d2d18 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:37.504385681 +0000 UTC m=+33.132538361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret") pod "global-pull-secret-syncer-l8zcg" (UID: "56c38f48-b836-41fb-92a1-2190725d2d18") : object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:21.504534 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.504417    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:37.504409909 +0000 UTC m=+33.132562590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:21.810679 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:21.810583    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:21.810679 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:21.810631    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:21.810912 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.810699    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:21.810912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:21.810733    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:21.810912 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.810837    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:21.811096 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:21.810936    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:23.810414 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:23.810384    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:23.810797 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:23.810478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:23.810797 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:23.810482    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:23.810797 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:23.810491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:23.810797 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:23.810574    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:23.810797 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:23.810650    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:25.811077 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:25.810719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:25.811426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:25.810763    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:25.811426 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:25.811130    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:25.811426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:25.810763    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:25.811426 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:25.811186    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:25.811426 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:25.811249    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:26.018568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.018534    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf0778c3-c4ce-42d6-b7e0-51544b593949" containerID="5a6208fa9d15cd56e9fdb6f7a9d44b0ae0d791d9938faa82f2a0f1c1409656ee" exitCode=0
Sep 09 17:17:26.018815 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.018610    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerDied","Data":"5a6208fa9d15cd56e9fdb6f7a9d44b0ae0d791d9938faa82f2a0f1c1409656ee"}
Sep 09 17:17:26.020255 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.020232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rqqmz" event={"ID":"238714e6-85d5-4af3-9cb8-d4f87ef2b4de","Type":"ContainerStarted","Data":"8f9af4fde670aa6f9e5e100856a5ca282fcbfffff4f00da4bed886f6e88443de"}
Sep 09 17:17:26.021820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.021803    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q" event={"ID":"790c0016-2adc-467b-9a36-9def50ac5db4","Type":"ContainerStarted","Data":"147b8f68cfef39b1219da6a2a5f805ba92bce2779a6678bf5546f12826bb80a7"}
Sep 09 17:17:26.023285 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.023256    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pdwrx" event={"ID":"849b01c7-caef-49a5-b7e2-b2274e4ff53b","Type":"ContainerStarted","Data":"4e953ab9ef6ac4277aecee5fb6c3a1fbfe6dc9506dbb1ef771bebdb594bb44d7"}
Sep 09 17:17:26.024741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.024716    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qngb4" event={"ID":"19a374f5-cf67-4db9-bd80-518a808fcd26","Type":"ContainerStarted","Data":"0625d57f38f91f6c001afb48563e7a612a27740dfe60b025efde22614ee81937"}
Sep 09 17:17:26.026129 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.026105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" event={"ID":"a63bc5640b9cbe7de0e68e356f93dd93","Type":"ContainerStarted","Data":"36711e49adbcae4f6da2538e59d502d07dc270400a93a5b78478648efb72c2a2"}
Sep 09 17:17:26.027580 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.027553    2558 generic.go:356] "Generic (PLEG): container finished" podID="9ad0f471830ea8bb8eada66ef51ef7e5" containerID="bb6999822cb983a0c98bf560d4d067bbad425313c56c0b39acee85ca49b284f6" exitCode=0
Sep 09 17:17:26.027654 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.027616    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"9ad0f471830ea8bb8eada66ef51ef7e5","Type":"ContainerDied","Data":"bb6999822cb983a0c98bf560d4d067bbad425313c56c0b39acee85ca49b284f6"}
Sep 09 17:17:26.028939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.028920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qpr8j" event={"ID":"b07f3322-92ad-4575-af6a-ae77de513515","Type":"ContainerStarted","Data":"e432b4ee8718f5d18eca5c7ce84c8f745ed3dd518151f0b7cbdd07ceab635697"}
Sep 09 17:17:26.030281 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.030263    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c84jg" event={"ID":"5a428b72-d41b-4a99-a171-75bfccbbe1b9","Type":"ContainerStarted","Data":"caec6b5800aabcc289cd0012eb1a2fb919e17d0d074af1889bb9fdcde0cbd46e"}
Sep 09 17:17:26.032801 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.032785    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:17:26.033127 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.033109    2558 generic.go:356] "Generic (PLEG): container finished" podID="589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8" containerID="7b1b70cff665a87c69a8e27b611912c5b46fae60cfd4f5505499cfc06822f233" exitCode=1
Sep 09 17:17:26.033186 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.033135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"34956a09991099834a25f013897ef9665278f69ff163cf2984fbb2817cc2cc3d"}
Sep 09 17:17:26.033186 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.033178    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"60cb97e34d340b784fba8d529594f7fda9aa4ff8ee27d686cb773a3b1a83a32f"}
Sep 09 17:17:26.033247 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.033206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"4842a5d822e01b64f93ecd6f4ffb0f55e9296f125297f0c838a7f402cc53bc5f"}
Sep 09 17:17:26.033247 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.033233    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"e8ac2c6843b51f957b5a68e3b4658c3e9eb354f5f77bb82bbb91227d4438998b"}
Sep 09 17:17:26.033305 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.033260    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerDied","Data":"7b1b70cff665a87c69a8e27b611912c5b46fae60cfd4f5505499cfc06822f233"}
Sep 09 17:17:26.033305 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.033288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"435f53a1eefd233913f0ea4abf1cc80d14a089125eceaa92f27de2332afcae72"}
Sep 09 17:17:26.054478 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.054416    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rqqmz" podStartSLOduration=3.132738584 podStartE2EDuration="22.05440336s" totalImagesPullingTime="18.921664776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:26.054094938 +0000 UTC m=+21.682247639" watchObservedRunningTime="2026-09-09 17:17:26.05440336 +0000 UTC m=+21.682556088"
Sep 09 17:17:26.070391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.070300    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qpr8j" podStartSLOduration=2.181754889 podStartE2EDuration="21.070289592s" totalImagesPullingTime="18.888534703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:26.069887549 +0000 UTC m=+21.698040253" watchObservedRunningTime="2026-09-09 17:17:26.070289592 +0000 UTC m=+21.698442322"
Sep 09 17:17:26.086295 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.084334    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-100.ec2.internal" podStartSLOduration=21.084314447 podStartE2EDuration="21.084314447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:26.083403551 +0000 UTC m=+21.711556255" watchObservedRunningTime="2026-09-09 17:17:26.084314447 +0000 UTC m=+21.712467149"
Sep 09 17:17:26.098115 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.098077    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qngb4" podStartSLOduration=3.146840362 podStartE2EDuration="22.098062466s" totalImagesPullingTime="18.951222104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:26.097813288 +0000 UTC m=+21.725965985" watchObservedRunningTime="2026-09-09 17:17:26.098062466 +0000 UTC m=+21.726215167"
Sep 09 17:17:26.111330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.111282    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pdwrx" podStartSLOduration=3.154330361 podStartE2EDuration="22.111270139s" totalImagesPullingTime="18.956939778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:26.111131362 +0000 UTC m=+21.739284064" watchObservedRunningTime="2026-09-09 17:17:26.111270139 +0000 UTC m=+21.739422842"
Sep 09 17:17:26.127867 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.127827    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c84jg" podStartSLOduration=2.8577068089999997 podStartE2EDuration="22.127813917s" totalImagesPullingTime="19.270107108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:26.127719251 +0000 UTC m=+21.755871965" watchObservedRunningTime="2026-09-09 17:17:26.127813917 +0000 UTC m=+21.755966619"
Sep 09 17:17:26.269743 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.269705    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qngb4_19a374f5-cf67-4db9-bd80-518a808fcd26/konnectivity-agent/0.log" limit=1024
Sep 09 17:17:26.335047 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.334944    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qngb4_19a374f5-cf67-4db9-bd80-518a808fcd26/konnectivity-agent/0.log" limit=1024
Sep 09 17:17:26.340074 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.340053    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:26.340533 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:26.340520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:27.037871 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.037811    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65xwz" event={"ID":"2896185f-281a-482f-a8ad-7a4895391831","Type":"ContainerStarted","Data":"28797db1ffe36e0edb23e6ad21bf969c639605a0875fdb0b2bde20eb57c7fa83"}
Sep 09 17:17:27.038644 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.038341    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:27.039167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.039119    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qngb4"
Sep 09 17:17:27.053348 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.053301    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-65xwz" podStartSLOduration=4.12593377 podStartE2EDuration="23.053286457s" totalImagesPullingTime="18.927352687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:27.052443305 +0000 UTC m=+22.680596022" watchObservedRunningTime="2026-09-09 17:17:27.053286457 +0000 UTC m=+22.681439158"
Sep 09 17:17:27.197903 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.197666    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qngb4_19a374f5-cf67-4db9-bd80-518a808fcd26/konnectivity-agent/0.log" limit=1024
Sep 09 17:17:27.293651 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.293518    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qngb4_19a374f5-cf67-4db9-bd80-518a808fcd26/konnectivity-agent/0.log" limit=1024
Sep 09 17:17:27.487262 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.487239    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 17:17:27.738622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.738500    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T17:17:27.487257189Z","UUID":"9bff20f3-d0dc-44d1-9f7c-bdd3beed180f","Handler":null,"Name":"","Endpoint":""}
Sep 09 17:17:27.740399 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.740377    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 17:17:27.740529 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.740411    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 17:17:27.810484 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.810460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:27.810616 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.810571    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:27.810616 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:27.810583    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:27.810726 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:27.810695    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:27.810800 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:27.810775    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:27.810904 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:27.810880    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:28.043216 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:28.043141    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:17:28.043603 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:28.043522    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"5f2cd85afd7c0549da405fc7862fb2c5934aace079edf01e93b77dbfd197a575"}
Sep 09 17:17:28.045158 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:28.045134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q" event={"ID":"790c0016-2adc-467b-9a36-9def50ac5db4","Type":"ContainerStarted","Data":"7af3a8af1c247a037e6d11389264b1c59af636113073ec190dcccf05c34a4fc0"}
Sep 09 17:17:28.046760 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:28.046738    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" event={"ID":"9ad0f471830ea8bb8eada66ef51ef7e5","Type":"ContainerStarted","Data":"57e55dae61be8c90025944eb6d489125a7e9265f04f9eddb784989e3ce2b8a62"}
Sep 09 17:17:28.061186 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:28.061136    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-100.ec2.internal" podStartSLOduration=23.061117678 podStartE2EDuration="23.061117678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:28.0609203 +0000 UTC m=+23.689073001" watchObservedRunningTime="2026-09-09 17:17:28.061117678 +0000 UTC m=+23.689270380"
Sep 09 17:17:28.445679 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:28.445642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qngb4_19a374f5-cf67-4db9-bd80-518a808fcd26/konnectivity-agent/0.log" limit=1024
Sep 09 17:17:28.498715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:28.498677    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qngb4_19a374f5-cf67-4db9-bd80-518a808fcd26/konnectivity-agent/0.log" limit=1024
Sep 09 17:17:29.052483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:29.052410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q" event={"ID":"790c0016-2adc-467b-9a36-9def50ac5db4","Type":"ContainerStarted","Data":"f19a3e7cbc7a9688b7e7a610fa205f18393964ff75db674cee74a621a19ca7f8"}
Sep 09 17:17:29.071371 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:29.071329    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fg48q" podStartSLOduration=3.369113267 podStartE2EDuration="25.07131422s" totalImagesPullingTime="21.702200953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:29.071230658 +0000 UTC m=+24.699383364" watchObservedRunningTime="2026-09-09 17:17:29.07131422 +0000 UTC m=+24.699466922"
Sep 09 17:17:29.811139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:29.811101    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:29.811358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:29.811149    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:29.811358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:29.811150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:29.811358 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:29.811231    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:29.811358 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:29.811330    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:29.811583 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:29.811402    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:31.062719 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.062519    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf0778c3-c4ce-42d6-b7e0-51544b593949" containerID="61bbe3213467179edb9ab3b090dffef2750e1699e1373d7774d5bced625b16c2" exitCode=0
Sep 09 17:17:31.063351 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.062602    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerDied","Data":"61bbe3213467179edb9ab3b090dffef2750e1699e1373d7774d5bced625b16c2"}
Sep 09 17:17:31.066583 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.066565    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:17:31.066919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.066887    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"1de1fbd4b711cad0a1d60eb59a54025a8a436e68d3cbd779b23591e8708e9793"}
Sep 09 17:17:31.067178 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.067157    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:31.067271 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.067205    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:31.067330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.067313    2558 scope.go:117] "RemoveContainer" containerID="7b1b70cff665a87c69a8e27b611912c5b46fae60cfd4f5505499cfc06822f233"
Sep 09 17:17:31.082911 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.082886    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:31.810720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.810684    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:31.811031 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.810687    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:31.811031 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:31.810812    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:31.811031 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:31.810686    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:31.811031 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:31.810912    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:31.811031 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:31.811014    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:32.077642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.077551    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:17:32.078333 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.078000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" event={"ID":"589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8","Type":"ContainerStarted","Data":"f15ded2fbc5ea87e1c7f1e5af10c08b640e48e4daff7a58fb60e3937a912cc3b"}
Sep 09 17:17:32.078704 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.078624    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:32.099297 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.099267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:17:32.107513 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.107455    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8" podStartSLOduration=9.060527864 podStartE2EDuration="28.107437542s" totalImagesPullingTime="19.046909678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:32.107043261 +0000 UTC m=+27.735195965" watchObservedRunningTime="2026-09-09 17:17:32.107437542 +0000 UTC m=+27.735590281"
Sep 09 17:17:32.547944 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.547687    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l8zcg"]
Sep 09 17:17:32.548126 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.548108    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:32.548249 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:32.548228    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:32.548657 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.548515    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zmm4h"]
Sep 09 17:17:32.548657 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.548624    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:32.548821 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:32.548711    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:32.549271 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.549249    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tv95p"]
Sep 09 17:17:32.549383 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:32.549343    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:32.549442 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:32.549418    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:33.082198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:33.082168    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf0778c3-c4ce-42d6-b7e0-51544b593949" containerID="a89a94153f40fcf3e1da41b20481b113ae9bdb8fb999168439b522df9e80e3ee" exitCode=0
Sep 09 17:17:33.082587 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:33.082238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerDied","Data":"a89a94153f40fcf3e1da41b20481b113ae9bdb8fb999168439b522df9e80e3ee"}
Sep 09 17:17:33.810856 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:33.810774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:33.811041 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:33.810880    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:33.811041 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:33.810941    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:33.811148 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:33.811068    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:34.811594 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:34.811563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:34.811992 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:34.811695    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:35.088941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:35.088855    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf0778c3-c4ce-42d6-b7e0-51544b593949" containerID="7c4f7d1db43c97e54ce405911df58a4edb553978aff8439419b82895f1f82c3a" exitCode=0
Sep 09 17:17:35.088941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:35.088905    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerDied","Data":"7c4f7d1db43c97e54ce405911df58a4edb553978aff8439419b82895f1f82c3a"}
Sep 09 17:17:35.810521 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:35.810485    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:35.810521 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:35.810508    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:35.810728 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:35.810595    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:35.810767 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:35.810721    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:36.810971 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:36.810913    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:36.811399 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:36.811063    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l8zcg" podUID="56c38f48-b836-41fb-92a1-2190725d2d18"
Sep 09 17:17:37.421056 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:37.421019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:37.421354 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.421148    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 17:17:37.421354 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.421169    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 17:17:37.421354 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.421180    2558 projected.go:196] Error preparing data for projected volume kube-api-access-86gqv for pod openshift-network-diagnostics/network-check-target-tv95p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:37.421354 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.421233    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv podName:042adbf6-fefd-465e-a52c-373c0491c7ba nodeName:}" failed. No retries permitted until 2026-09-09 17:18:09.421219957 +0000 UTC m=+65.049372637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-86gqv" (UniqueName: "kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv") pod "network-check-target-tv95p" (UID: "042adbf6-fefd-465e-a52c-373c0491c7ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 17:17:37.522083 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:37.522052    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:37.522781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:37.522106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:37.522781 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.522245    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:37.522781 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.522277    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:37.522781 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.522318    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret podName:56c38f48-b836-41fb-92a1-2190725d2d18 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:09.522299036 +0000 UTC m=+65.150451718 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret") pod "global-pull-secret-syncer-l8zcg" (UID: "56c38f48-b836-41fb-92a1-2190725d2d18") : object "kube-system"/"original-pull-secret" not registered
Sep 09 17:17:37.522781 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.522338    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:09.522329802 +0000 UTC m=+65.150482489 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 17:17:37.810939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:37.810910    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:37.811357 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:37.810973    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:37.811357 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.811103    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:17:37.811357 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:37.811211    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv95p" podUID="042adbf6-fefd-465e-a52c-373c0491c7ba"
Sep 09 17:17:38.119798 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.119723    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-100.ec2.internal" event="NodeReady"
Sep 09 17:17:38.119975 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.119891    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 17:17:38.176628 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.176597    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qdcbd"]
Sep 09 17:17:38.238329 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.238293    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hcclw"]
Sep 09 17:17:38.238486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.238421    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.241532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.241509    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 17:17:38.241759 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.241730    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 17:17:38.241982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.241949    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 17:17:38.242064 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.241949    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vg7zn\""
Sep 09 17:17:38.252843 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.252823    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qdcbd"]
Sep 09 17:17:38.252932 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.252859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hcclw"]
Sep 09 17:17:38.252995 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.252978    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.256008 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.255987    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 17:17:38.256727 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.256536    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 17:17:38.256727 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.256541    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cblwm\""
Sep 09 17:17:38.327808 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.327773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fch7d\" (UniqueName: \"kubernetes.io/projected/c13bf76c-fc53-4e46-a5ce-4a4b5e37c867-kube-api-access-fch7d\") pod \"ingress-canary-qdcbd\" (UID: \"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867\") " pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.328015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.327821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.328015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.327839    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef37b751-5814-4867-930a-9632adbcc0c0-tmp-dir\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.328015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.327865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c13bf76c-fc53-4e46-a5ce-4a4b5e37c867-cert\") pod \"ingress-canary-qdcbd\" (UID: \"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867\") " pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.328015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.327893    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnt6\" (UniqueName: \"kubernetes.io/projected/ef37b751-5814-4867-930a-9632adbcc0c0-kube-api-access-sdnt6\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.328015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.327910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef37b751-5814-4867-930a-9632adbcc0c0-config-volume\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.429134 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.429098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.429302 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.429144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef37b751-5814-4867-930a-9632adbcc0c0-tmp-dir\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.429302 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:38.429260    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:17:38.429413 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:38.429349    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:38.929327059 +0000 UTC m=+34.557479757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:17:38.429413 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.429374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c13bf76c-fc53-4e46-a5ce-4a4b5e37c867-cert\") pod \"ingress-canary-qdcbd\" (UID: \"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867\") " pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.429525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.429428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnt6\" (UniqueName: \"kubernetes.io/projected/ef37b751-5814-4867-930a-9632adbcc0c0-kube-api-access-sdnt6\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.429525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.429460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef37b751-5814-4867-930a-9632adbcc0c0-config-volume\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.429619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.429529    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fch7d\" (UniqueName: \"kubernetes.io/projected/c13bf76c-fc53-4e46-a5ce-4a4b5e37c867-kube-api-access-fch7d\") pod \"ingress-canary-qdcbd\" (UID: \"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867\") " pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.429767 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.429747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef37b751-5814-4867-930a-9632adbcc0c0-tmp-dir\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.430149 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.430125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef37b751-5814-4867-930a-9632adbcc0c0-config-volume\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.433990 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.433805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c13bf76c-fc53-4e46-a5ce-4a4b5e37c867-cert\") pod \"ingress-canary-qdcbd\" (UID: \"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867\") " pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.444934 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.444914    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fch7d\" (UniqueName: \"kubernetes.io/projected/c13bf76c-fc53-4e46-a5ce-4a4b5e37c867-kube-api-access-fch7d\") pod \"ingress-canary-qdcbd\" (UID: \"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867\") " pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.445033 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.444973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnt6\" (UniqueName: \"kubernetes.io/projected/ef37b751-5814-4867-930a-9632adbcc0c0-kube-api-access-sdnt6\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.552065 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.552030    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qdcbd"
Sep 09 17:17:38.811203 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.811123    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:17:38.814401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.814377    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 17:17:38.933322 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:38.933285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:38.933519 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:38.933459    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:17:38.933568 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:38.933551    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:39.933529007 +0000 UTC m=+35.561681739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:17:39.810373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.810340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:17:39.810553 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.810525    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:17:39.813350 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.813324    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 17:17:39.814252 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.814231    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kwqx\""
Sep 09 17:17:39.814252 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.814244    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 17:17:39.814554 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.814527    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qln5f\""
Sep 09 17:17:39.814653 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.814563    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 17:17:39.941716 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:39.941685    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:39.941865 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:39.941837    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:17:39.941918 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:39.941907    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:41.9418916 +0000 UTC m=+37.570044281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:17:40.235042 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:40.234992    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qdcbd"]
Sep 09 17:17:40.237889 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:17:40.237846    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13bf76c_fc53_4e46_a5ce_4a4b5e37c867.slice/crio-f8f92224c691a35e9f912ab8b77c3de0dacdc27718f98bb0c4777b8b66806eb0 WatchSource:0}: Error finding container f8f92224c691a35e9f912ab8b77c3de0dacdc27718f98bb0c4777b8b66806eb0: Status 404 returned error can't find the container with id f8f92224c691a35e9f912ab8b77c3de0dacdc27718f98bb0c4777b8b66806eb0
Sep 09 17:17:41.104721 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:41.104687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qdcbd" event={"ID":"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867","Type":"ContainerStarted","Data":"f8f92224c691a35e9f912ab8b77c3de0dacdc27718f98bb0c4777b8b66806eb0"}
Sep 09 17:17:41.108108 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:41.108079    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf0778c3-c4ce-42d6-b7e0-51544b593949" containerID="b96946993f1d7c2bc470494ba92d5cdf822ab38f3666441d2dadb889a5936b11" exitCode=0
Sep 09 17:17:41.108219 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:41.108124    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerDied","Data":"b96946993f1d7c2bc470494ba92d5cdf822ab38f3666441d2dadb889a5936b11"}
Sep 09 17:17:41.957656 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:41.957431    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:41.957802 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:41.957630    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:17:41.957802 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:41.957775    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:45.957754006 +0000 UTC m=+41.585906688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:17:42.114409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:42.114376    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf0778c3-c4ce-42d6-b7e0-51544b593949" containerID="1eeb484064e759cc197bd04835794bb3c198b191a697c88e15aea5ffc0d5a990" exitCode=0
Sep 09 17:17:42.114844 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:42.114428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerDied","Data":"1eeb484064e759cc197bd04835794bb3c198b191a697c88e15aea5ffc0d5a990"}
Sep 09 17:17:43.120427 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:43.120399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptt54" event={"ID":"cf0778c3-c4ce-42d6-b7e0-51544b593949","Type":"ContainerStarted","Data":"cdfbb808b8d64021561a0472f544b9ccc222f1e637bd51c0f2b025b10ac535c0"}
Sep 09 17:17:43.140447 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:43.140408    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ptt54" podStartSLOduration=8.191127991 podStartE2EDuration="39.140393551s" totalImagesPullingTime="30.94926556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:43.139223714 +0000 UTC m=+38.767376430" watchObservedRunningTime="2026-09-09 17:17:43.140393551 +0000 UTC m=+38.768546252"
Sep 09 17:17:44.124815 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:44.124781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qdcbd" event={"ID":"c13bf76c-fc53-4e46-a5ce-4a4b5e37c867","Type":"ContainerStarted","Data":"81833631769c42405a652e1718081dc315cd4d8e80ed0269aa55761e22586f2d"}
Sep 09 17:17:44.139864 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:44.139815    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qdcbd" podStartSLOduration=3.332278879 podStartE2EDuration="6.139795008s" totalImagesPullingTime="2.807516129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:44.139122714 +0000 UTC m=+39.767275428" watchObservedRunningTime="2026-09-09 17:17:44.139795008 +0000 UTC m=+39.767947713"
Sep 09 17:17:45.987575 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:45.987528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:45.988082 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:45.987703    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:17:45.988082 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:45.987787    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:53.987766112 +0000 UTC m=+49.615918810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:17:54.044367 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:17:54.044325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:17:54.044779 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:54.044478    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:17:54.044779 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:17:54.044548    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:10.044532571 +0000 UTC m=+65.672685256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:18:04.094750 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:04.094716    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snbz8"
Sep 09 17:18:07.515373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.515340    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"]
Sep 09 17:18:07.518039 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.518016    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.518722 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.518701    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"]
Sep 09 17:18:07.520348 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.520331    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.520794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.520770    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 17:18:07.520794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.520785    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 17:18:07.522476 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.522439    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-87ft8\""
Sep 09 17:18:07.522476 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.522468    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 17:18:07.522621 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.522553    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 17:18:07.523672 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.523542    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 17:18:07.524309 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.523976    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 17:18:07.524309 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.524125    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 17:18:07.524309 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.524254    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 17:18:07.527791 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.527767    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"]
Sep 09 17:18:07.532423 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.532404    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"]
Sep 09 17:18:07.642208 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642177    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.642208 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/319267dd-5f0d-42db-a56c-327cd573c9fb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.642409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642231    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.642409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snm76\" (UniqueName: \"kubernetes.io/projected/319267dd-5f0d-42db-a56c-327cd573c9fb-kube-api-access-snm76\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.642409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208e2c8f-35e2-48c5-8b93-fccf9d2e5280-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9\" (UID: \"208e2c8f-35e2-48c5-8b93-fccf9d2e5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.642409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642380    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-hub\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.642596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-ca\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.642596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.642521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzpk\" (UniqueName: \"kubernetes.io/projected/208e2c8f-35e2-48c5-8b93-fccf9d2e5280-kube-api-access-kmzpk\") pod \"managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9\" (UID: \"208e2c8f-35e2-48c5-8b93-fccf9d2e5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.743042 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.742994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-ca\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.743042 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743046    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzpk\" (UniqueName: \"kubernetes.io/projected/208e2c8f-35e2-48c5-8b93-fccf9d2e5280-kube-api-access-kmzpk\") pod \"managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9\" (UID: \"208e2c8f-35e2-48c5-8b93-fccf9d2e5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.743219 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743078    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.743219 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/319267dd-5f0d-42db-a56c-327cd573c9fb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.743219 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.743219 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snm76\" (UniqueName: \"kubernetes.io/projected/319267dd-5f0d-42db-a56c-327cd573c9fb-kube-api-access-snm76\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.743394 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208e2c8f-35e2-48c5-8b93-fccf9d2e5280-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9\" (UID: \"208e2c8f-35e2-48c5-8b93-fccf9d2e5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.743394 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-hub\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.743891 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.743864    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/319267dd-5f0d-42db-a56c-327cd573c9fb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.746486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.746464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-hub\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.746648 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.746627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-ca\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.746689 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.746634    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.746689 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.746677    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/319267dd-5f0d-42db-a56c-327cd573c9fb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.746751 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.746685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208e2c8f-35e2-48c5-8b93-fccf9d2e5280-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9\" (UID: \"208e2c8f-35e2-48c5-8b93-fccf9d2e5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.750794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.750766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snm76\" (UniqueName: \"kubernetes.io/projected/319267dd-5f0d-42db-a56c-327cd573c9fb-kube-api-access-snm76\") pod \"cluster-proxy-proxy-agent-59d87c9599-69nqb\" (UID: \"319267dd-5f0d-42db-a56c-327cd573c9fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:07.751056 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.751035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzpk\" (UniqueName: \"kubernetes.io/projected/208e2c8f-35e2-48c5-8b93-fccf9d2e5280-kube-api-access-kmzpk\") pod \"managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9\" (UID: \"208e2c8f-35e2-48c5-8b93-fccf9d2e5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.843317 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.843223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"
Sep 09 17:18:07.848542 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:07.848517    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:18:08.117305 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:08.117233    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9"]
Sep 09 17:18:08.120340 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:18:08.120312    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod208e2c8f_35e2_48c5_8b93_fccf9d2e5280.slice/crio-3ad8e8a8470670ff88a0d061a77f16c3b26a274d29a95bd5ba71075b9659c94d WatchSource:0}: Error finding container 3ad8e8a8470670ff88a0d061a77f16c3b26a274d29a95bd5ba71075b9659c94d: Status 404 returned error can't find the container with id 3ad8e8a8470670ff88a0d061a77f16c3b26a274d29a95bd5ba71075b9659c94d
Sep 09 17:18:08.120577 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:08.120542    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"]
Sep 09 17:18:08.123434 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:18:08.123414    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod319267dd_5f0d_42db_a56c_327cd573c9fb.slice/crio-4efa192252862600de2d058c678300f8e0fb9a28af737ac68e12d51b82920324 WatchSource:0}: Error finding container 4efa192252862600de2d058c678300f8e0fb9a28af737ac68e12d51b82920324: Status 404 returned error can't find the container with id 4efa192252862600de2d058c678300f8e0fb9a28af737ac68e12d51b82920324
Sep 09 17:18:08.182590 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:08.182554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" event={"ID":"319267dd-5f0d-42db-a56c-327cd573c9fb","Type":"ContainerStarted","Data":"4efa192252862600de2d058c678300f8e0fb9a28af737ac68e12d51b82920324"}
Sep 09 17:18:08.183539 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:08.183513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9" event={"ID":"208e2c8f-35e2-48c5-8b93-fccf9d2e5280","Type":"ContainerStarted","Data":"3ad8e8a8470670ff88a0d061a77f16c3b26a274d29a95bd5ba71075b9659c94d"}
Sep 09 17:18:09.457908 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.457863    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:18:09.463485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.463455    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 17:18:09.470965 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.470927    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 17:18:09.482118 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.482093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gqv\" (UniqueName: \"kubernetes.io/projected/042adbf6-fefd-465e-a52c-373c0491c7ba-kube-api-access-86gqv\") pod \"network-check-target-tv95p\" (UID: \"042adbf6-fefd-465e-a52c-373c0491c7ba\") " pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:18:09.529187 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.529153    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qln5f\""
Sep 09 17:18:09.537558 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.537523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:18:09.558708 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.558524    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:18:09.558708 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.558598    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:18:09.561330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.561301    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 17:18:09.561464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.561359    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 17:18:09.568832 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:18:09.568806    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 17:18:09.568944 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:18:09.568876    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:13.568855207 +0000 UTC m=+129.197007888 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : secret "metrics-daemon-secret" not found
Sep 09 17:18:09.572861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.572840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c38f48-b836-41fb-92a1-2190725d2d18-original-pull-secret\") pod \"global-pull-secret-syncer-l8zcg\" (UID: \"56c38f48-b836-41fb-92a1-2190725d2d18\") " pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:18:09.721664 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.721577    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l8zcg"
Sep 09 17:18:09.815415 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:09.815351    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tv95p"]
Sep 09 17:18:10.063151 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:10.063065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:18:10.063294 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:18:10.063227    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:18:10.063350 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:18:10.063305    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:42.063281971 +0000 UTC m=+97.691434652 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:18:10.628397 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:18:10.628361    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042adbf6_fefd_465e_a52c_373c0491c7ba.slice/crio-79688aa1f584036bc3a387645685edbfdbcdf659bedba28392bb8d420b2154bc WatchSource:0}: Error finding container 79688aa1f584036bc3a387645685edbfdbcdf659bedba28392bb8d420b2154bc: Status 404 returned error can't find the container with id 79688aa1f584036bc3a387645685edbfdbcdf659bedba28392bb8d420b2154bc
Sep 09 17:18:10.894788 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:10.894655    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l8zcg"]
Sep 09 17:18:10.898064 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:18:10.898009    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c38f48_b836_41fb_92a1_2190725d2d18.slice/crio-09473962135f70c6abdf1b659b946f97fa6272bbbcd7bf864269ee086830602d WatchSource:0}: Error finding container 09473962135f70c6abdf1b659b946f97fa6272bbbcd7bf864269ee086830602d: Status 404 returned error can't find the container with id 09473962135f70c6abdf1b659b946f97fa6272bbbcd7bf864269ee086830602d
Sep 09 17:18:11.194920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:11.194876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l8zcg" event={"ID":"56c38f48-b836-41fb-92a1-2190725d2d18","Type":"ContainerStarted","Data":"09473962135f70c6abdf1b659b946f97fa6272bbbcd7bf864269ee086830602d"}
Sep 09 17:18:11.196346 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:11.196317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tv95p" event={"ID":"042adbf6-fefd-465e-a52c-373c0491c7ba","Type":"ContainerStarted","Data":"79688aa1f584036bc3a387645685edbfdbcdf659bedba28392bb8d420b2154bc"}
Sep 09 17:18:11.198008 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:11.197982    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9" event={"ID":"208e2c8f-35e2-48c5-8b93-fccf9d2e5280","Type":"ContainerStarted","Data":"b20c96bc565f8430828d5c5d2044a3a51330f4be24c03f2b615cf5d1019ec9ea"}
Sep 09 17:18:11.212806 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:11.212752    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9" podStartSLOduration=1.676817542 podStartE2EDuration="4.212732678s" totalImagesPullingTime="2.535915136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:18:11.212251861 +0000 UTC m=+66.840404563" watchObservedRunningTime="2026-09-09 17:18:11.212732678 +0000 UTC m=+66.840885381"
Sep 09 17:18:17.226027 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:17.225989    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" event={"ID":"319267dd-5f0d-42db-a56c-327cd573c9fb","Type":"ContainerStarted","Data":"f4502f9937fde8b1adc99154a6435841f09d8755f23234c0dc624e0ddec3998d"}
Sep 09 17:18:17.226503 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:17.226046    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" event={"ID":"319267dd-5f0d-42db-a56c-327cd573c9fb","Type":"ContainerStarted","Data":"34e7af3391340b2dc27d86eb205a0dfb9715f1b4acfdf2be6d49ffd9ff96653b"}
Sep 09 17:18:17.227698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:17.227657    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l8zcg" event={"ID":"56c38f48-b836-41fb-92a1-2190725d2d18","Type":"ContainerStarted","Data":"e04d0afea8fd9f473fb57e8f39799c10f3c8333515b591f4bd536bd3976f0f02"}
Sep 09 17:18:17.230739 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:17.230687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tv95p" event={"ID":"042adbf6-fefd-465e-a52c-373c0491c7ba","Type":"ContainerStarted","Data":"db10b45578cb696b0f15bc973779184c35bd916d615eca289d3ac921dd4990c3"}
Sep 09 17:18:17.230867 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:17.230812    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:18:17.248616 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:17.248562    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l8zcg" podStartSLOduration=67.074972193 podStartE2EDuration="1m13.248541673s" totalImagesPullingTime="6.17356948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:18:17.248463121 +0000 UTC m=+72.876615824" watchObservedRunningTime="2026-09-09 17:18:17.248541673 +0000 UTC m=+72.876694378"
Sep 09 17:18:17.263969 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:17.263909    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tv95p" podStartSLOduration=66.834908717 podStartE2EDuration="1m13.26389233s" totalImagesPullingTime="6.428983613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:18:17.263514725 +0000 UTC m=+72.891667429" watchObservedRunningTime="2026-09-09 17:18:17.26389233 +0000 UTC m=+72.892045033"
Sep 09 17:18:18.237077 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:18.237044    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" event={"ID":"319267dd-5f0d-42db-a56c-327cd573c9fb","Type":"ContainerStarted","Data":"20a104161e4852170ff7012aa30600020ede503b1cbbdec969d73c200bda7b97"}
Sep 09 17:18:18.255848 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:18.255799    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" podStartSLOduration=2.305293183 podStartE2EDuration="11.25578118s" totalImagesPullingTime="8.950487997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:18:18.254999519 +0000 UTC m=+73.883152221" watchObservedRunningTime="2026-09-09 17:18:18.25578118 +0000 UTC m=+73.883933884"
Sep 09 17:18:42.083551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:42.083427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:18:42.083551 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:18:42.083538    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:18:42.084024 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:18:42.083601    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:46.083583782 +0000 UTC m=+161.711736476 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:18:48.239514 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:18:48.239480    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tv95p"
Sep 09 17:19:13.594631 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:19:13.594582    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:19:13.595139 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:19:13.594736    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 17:19:13.595139 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:19:13.594803    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs podName:92691a45-2af1-48ac-830a-ead94a5117e6 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:15.594785739 +0000 UTC m=+251.222938432 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs") pod "network-metrics-daemon-zmm4h" (UID: "92691a45-2af1-48ac-830a-ead94a5117e6") : secret "metrics-daemon-secret" not found
Sep 09 17:19:41.263278 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:19:41.263217    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hcclw" podUID="ef37b751-5814-4867-930a-9632adbcc0c0"
Sep 09 17:19:42.821302 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:19:42.821256    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zmm4h" podUID="92691a45-2af1-48ac-830a-ead94a5117e6"
Sep 09 17:19:46.117053 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:19:46.117006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:19:46.117634 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:19:46.117149    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 17:19:46.117634 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:19:46.117227    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls podName:ef37b751-5814-4867-930a-9632adbcc0c0 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:48.117207278 +0000 UTC m=+283.745359980 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls") pod "dns-default-hcclw" (UID: "ef37b751-5814-4867-930a-9632adbcc0c0") : secret "dns-default-metrics-tls" not found
Sep 09 17:19:54.811702 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:19:54.811661    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hcclw"
Sep 09 17:19:55.810257 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:19:55.810221    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:20:11.559711 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:11.559676    2558 generic.go:356] "Generic (PLEG): container finished" podID="208e2c8f-35e2-48c5-8b93-fccf9d2e5280" containerID="b20c96bc565f8430828d5c5d2044a3a51330f4be24c03f2b615cf5d1019ec9ea" exitCode=255
Sep 09 17:20:11.560131 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:11.559717    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9" event={"ID":"208e2c8f-35e2-48c5-8b93-fccf9d2e5280","Type":"ContainerDied","Data":"b20c96bc565f8430828d5c5d2044a3a51330f4be24c03f2b615cf5d1019ec9ea"}
Sep 09 17:20:11.560131 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:11.560002    2558 scope.go:117] "RemoveContainer" containerID="b20c96bc565f8430828d5c5d2044a3a51330f4be24c03f2b615cf5d1019ec9ea"
Sep 09 17:20:12.564145 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:12.564099    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f9c5954dd-g8mt9" event={"ID":"208e2c8f-35e2-48c5-8b93-fccf9d2e5280","Type":"ContainerStarted","Data":"28f62262b733f592899a0062246409feef33a1fab54f6022d3b9ee88125f86aa"}
Sep 09 17:20:56.557126 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.557089    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f5h2t"]
Sep 09 17:20:56.559009 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.558993    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.562063 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.562040    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 17:20:56.562182 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.562061    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 17:20:56.563215 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.563196    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 17:20:56.563215 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.563205    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wvqv8\""
Sep 09 17:20:56.563376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.563256    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 17:20:56.585019 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.584986    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5h2t"]
Sep 09 17:20:56.594335 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.594302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.594493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.594362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d03fbd8e-7c13-46fc-a575-812ba078b61d-data-volume\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.594493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.594426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dn7l\" (UniqueName: \"kubernetes.io/projected/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-api-access-6dn7l\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.594493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.594469    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d03fbd8e-7c13-46fc-a575-812ba078b61d-crio-socket\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.594685 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.594494    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.695148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.695096    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.695308 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:56.695234    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 17:20:56.695308 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.695258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d03fbd8e-7c13-46fc-a575-812ba078b61d-data-volume\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.695308 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:56.695297    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:20:57.195281007 +0000 UTC m=+232.823433701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : configmap "kube-rbac-proxy" not found
Sep 09 17:20:56.695420 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.695313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dn7l\" (UniqueName: \"kubernetes.io/projected/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-api-access-6dn7l\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.695420 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.695339    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d03fbd8e-7c13-46fc-a575-812ba078b61d-crio-socket\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.695420 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.695356    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.695524 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:56.695450    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 17:20:56.695524 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.695459    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d03fbd8e-7c13-46fc-a575-812ba078b61d-crio-socket\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.695524 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:56.695486    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:20:57.195474389 +0000 UTC m=+232.823627071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : secret "insights-runtime-extractor-tls" not found
Sep 09 17:20:56.695657 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.695595    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d03fbd8e-7c13-46fc-a575-812ba078b61d-data-volume\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:56.703022 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:56.702996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dn7l\" (UniqueName: \"kubernetes.io/projected/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-api-access-6dn7l\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:57.198634 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:57.198599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:57.198812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:57.198654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:57.198812 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:57.198730    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 17:20:57.198812 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:57.198755    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 17:20:57.198812 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:57.198808    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:20:58.198795009 +0000 UTC m=+233.826947689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : secret "insights-runtime-extractor-tls" not found
Sep 09 17:20:57.198938 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:57.198820    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:20:58.198814128 +0000 UTC m=+233.826966808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : configmap "kube-rbac-proxy" not found
Sep 09 17:20:58.205446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:58.205403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:58.205864 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:58.205465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:20:58.205864 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:58.205530    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 17:20:58.205864 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:58.205569    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 17:20:58.205864 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:58.205603    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:21:00.205586386 +0000 UTC m=+235.833739066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : configmap "kube-rbac-proxy" not found
Sep 09 17:20:58.205864 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:20:58.205618    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:21:00.205611546 +0000 UTC m=+235.833764227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : secret "insights-runtime-extractor-tls" not found
Sep 09 17:20:58.735619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:58.735588    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pdwrx_849b01c7-caef-49a5-b7e2-b2274e4ff53b/dns-node-resolver/0.log"
Sep 09 17:20:59.335740 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:20:59.335716    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qpr8j_b07f3322-92ad-4575-af6a-ae77de513515/node-ca/0.log"
Sep 09 17:21:00.218613 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:00.218559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:00.218798 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:00.218631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:00.218798 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:21:00.218695    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 17:21:00.218798 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:21:00.218717    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 17:21:00.218798 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:21:00.218772    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:21:04.218755775 +0000 UTC m=+239.846908455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : configmap "kube-rbac-proxy" not found
Sep 09 17:21:00.218798 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:21:00.218787    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:21:04.218781867 +0000 UTC m=+239.846934547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : secret "insights-runtime-extractor-tls" not found
Sep 09 17:21:00.536109 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:00.536031    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qdcbd_c13bf76c-fc53-4e46-a5ce-4a4b5e37c867/serve-healthcheck-canary/0.log"
Sep 09 17:21:04.248773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:04.248715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:04.249272 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:04.248792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:04.249272 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:21:04.248877    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 17:21:04.249272 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:21:04.248984    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls podName:d03fbd8e-7c13-46fc-a575-812ba078b61d nodeName:}" failed. No retries permitted until 2026-09-09 17:21:12.248930506 +0000 UTC m=+247.877083187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5h2t" (UID: "d03fbd8e-7c13-46fc-a575-812ba078b61d") : secret "insights-runtime-extractor-tls" not found
Sep 09 17:21:04.249387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:04.249331    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03fbd8e-7c13-46fc-a575-812ba078b61d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:12.308255 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:12.308218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:12.310556 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:12.310528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03fbd8e-7c13-46fc-a575-812ba078b61d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5h2t\" (UID: \"d03fbd8e-7c13-46fc-a575-812ba078b61d\") " pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:12.471602 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:12.471570    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wvqv8\""
Sep 09 17:21:12.479767 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:12.479742    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5h2t"
Sep 09 17:21:12.748121 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:12.748088    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5h2t"]
Sep 09 17:21:12.751845 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:21:12.751818    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03fbd8e_7c13_46fc_a575_812ba078b61d.slice/crio-c47bbcee4e79f0e007e531c41445d6c41d1c6be38a8a49bb1a19ce3c5ba176f7 WatchSource:0}: Error finding container c47bbcee4e79f0e007e531c41445d6c41d1c6be38a8a49bb1a19ce3c5ba176f7: Status 404 returned error can't find the container with id c47bbcee4e79f0e007e531c41445d6c41d1c6be38a8a49bb1a19ce3c5ba176f7
Sep 09 17:21:13.741496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:13.741456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5h2t" event={"ID":"d03fbd8e-7c13-46fc-a575-812ba078b61d","Type":"ContainerStarted","Data":"5d98ec916ed7f42b8696d6d087fb62fe7d71d3bab29ccd2b249c6490c95167b0"}
Sep 09 17:21:13.741834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:13.741518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5h2t" event={"ID":"d03fbd8e-7c13-46fc-a575-812ba078b61d","Type":"ContainerStarted","Data":"b3b34966873ae0ff92f181b4f9d5f2e63cd3e292ffd7c7565fb60df6c29643fa"}
Sep 09 17:21:13.741834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:13.741536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5h2t" event={"ID":"d03fbd8e-7c13-46fc-a575-812ba078b61d","Type":"ContainerStarted","Data":"c47bbcee4e79f0e007e531c41445d6c41d1c6be38a8a49bb1a19ce3c5ba176f7"}
Sep 09 17:21:15.633166 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:15.633126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:21:15.635417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:15.635382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92691a45-2af1-48ac-830a-ead94a5117e6-metrics-certs\") pod \"network-metrics-daemon-zmm4h\" (UID: \"92691a45-2af1-48ac-830a-ead94a5117e6\") " pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:21:15.749234 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:15.749197    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5h2t" event={"ID":"d03fbd8e-7c13-46fc-a575-812ba078b61d","Type":"ContainerStarted","Data":"54db669eab40f76507390a9364cbdddfe022716715c13bb051bc0a08314fb6a6"}
Sep 09 17:21:15.766341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:15.766291    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f5h2t" podStartSLOduration=17.723168908 podStartE2EDuration="19.766273976s" totalImagesPullingTime="2.043105068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:20:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:15.765906139 +0000 UTC m=+251.394058854" watchObservedRunningTime="2026-09-09 17:21:15.766273976 +0000 UTC m=+251.394426679"
Sep 09 17:21:15.913638 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:15.913552    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kwqx\""
Sep 09 17:21:15.921749 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:15.921719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmm4h"
Sep 09 17:21:16.190328 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:16.190297    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zmm4h"]
Sep 09 17:21:16.193665 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:21:16.193640    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92691a45_2af1_48ac_830a_ead94a5117e6.slice/crio-6155c61409c52bd02988ef3867a982d42a594bda3a2adcb8bf0e324a447899dd WatchSource:0}: Error finding container 6155c61409c52bd02988ef3867a982d42a594bda3a2adcb8bf0e324a447899dd: Status 404 returned error can't find the container with id 6155c61409c52bd02988ef3867a982d42a594bda3a2adcb8bf0e324a447899dd
Sep 09 17:21:16.754415 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:16.754382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmm4h" event={"ID":"92691a45-2af1-48ac-830a-ead94a5117e6","Type":"ContainerStarted","Data":"6155c61409c52bd02988ef3867a982d42a594bda3a2adcb8bf0e324a447899dd"}
Sep 09 17:21:17.758913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:17.758878    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmm4h" event={"ID":"92691a45-2af1-48ac-830a-ead94a5117e6","Type":"ContainerStarted","Data":"e5cebec7055184fb356336bb4b9ce5f70224577eab53ea6b74f2452ea704e9ab"}
Sep 09 17:21:17.759327 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:17.758934    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmm4h" event={"ID":"92691a45-2af1-48ac-830a-ead94a5117e6","Type":"ContainerStarted","Data":"fcba2891b21d30433046dac30e67fc3da03b2b250f68809ff606524f8f289966"}
Sep 09 17:21:17.774254 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:17.774210    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zmm4h" podStartSLOduration=252.817359726 podStartE2EDuration="4m13.774193127s" totalImagesPullingTime="956.833401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:17.77291295 +0000 UTC m=+253.401065686" watchObservedRunningTime="2026-09-09 17:21:17.774193127 +0000 UTC m=+253.402345829"
Sep 09 17:21:37.768350 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.768277    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c8b66885-x8xhk"]
Sep 09 17:21:37.770635 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.770620    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.773747 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.773704    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 17:21:37.774050 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.773996    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 17:21:37.774287 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.774261    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 17:21:37.775508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.775062    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 17:21:37.775508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.775204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 17:21:37.775508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.775264    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 17:21:37.775508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.775382    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 17:21:37.776797 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.776774    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rfpcp\""
Sep 09 17:21:37.780108 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.779812    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 17:21:37.781642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.781622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8b66885-x8xhk"]
Sep 09 17:21:37.887487 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.887445    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-service-ca\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.887487 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.887481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-oauth-serving-cert\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.887697 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.887499    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-trusted-ca-bundle\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.887697 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.887515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9lb\" (UniqueName: \"kubernetes.io/projected/7e737cf2-0ee7-42b0-8529-ff441152ceed-kube-api-access-hx9lb\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.887697 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.887534    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-serving-cert\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.887697 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.887550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-config\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.887697 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.887655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-oauth-config\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.988992 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.988929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-oauth-serving-cert\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.988992 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.988993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-trusted-ca-bundle\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989207 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989009    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9lb\" (UniqueName: \"kubernetes.io/projected/7e737cf2-0ee7-42b0-8529-ff441152ceed-kube-api-access-hx9lb\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989207 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989030    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-serving-cert\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989207 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989045    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-config\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-oauth-config\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-service-ca\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989755 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-oauth-serving-cert\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989868 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-service-ca\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989929 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989849    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-config\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.989929 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.989856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-trusted-ca-bundle\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.991493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.991462    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-serving-cert\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.991579 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.991470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-oauth-config\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:37.996917 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:37.996899    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9lb\" (UniqueName: \"kubernetes.io/projected/7e737cf2-0ee7-42b0-8529-ff441152ceed-kube-api-access-hx9lb\") pod \"console-5c8b66885-x8xhk\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") " pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:38.083015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.082923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:38.366593 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.366519    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c8b66885-x8xhk"]
Sep 09 17:21:38.369738 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:21:38.369706    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e737cf2_0ee7_42b0_8529_ff441152ceed.slice/crio-c0a5bb743593682cb916dfbbb90b07601cf69bb302e16ad5008b9db98d32d696 WatchSource:0}: Error finding container c0a5bb743593682cb916dfbbb90b07601cf69bb302e16ad5008b9db98d32d696: Status 404 returned error can't find the container with id c0a5bb743593682cb916dfbbb90b07601cf69bb302e16ad5008b9db98d32d696
Sep 09 17:21:38.820011 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.819973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8b66885-x8xhk" event={"ID":"7e737cf2-0ee7-42b0-8529-ff441152ceed","Type":"ContainerStarted","Data":"c0a5bb743593682cb916dfbbb90b07601cf69bb302e16ad5008b9db98d32d696"}
Sep 09 17:21:38.980707 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.980673    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"]
Sep 09 17:21:38.982908 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.982893    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:38.986684 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.986659    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 17:21:38.986684 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.986659    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gt7kk\""
Sep 09 17:21:38.986869 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.986703    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 17:21:38.986869 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.986754    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 17:21:38.986869 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.986777    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 17:21:38.986869 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.986704    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 17:21:38.991607 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:38.991586    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"]
Sep 09 17:21:39.097270 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.097156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fec38056-a2ad-4785-ac80-69dc61eb7d17-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.097270 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.097226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec38056-a2ad-4785-ac80-69dc61eb7d17-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.097476 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.097305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fec38056-a2ad-4785-ac80-69dc61eb7d17-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.097476 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.097349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjnv\" (UniqueName: \"kubernetes.io/projected/fec38056-a2ad-4785-ac80-69dc61eb7d17-kube-api-access-vmjnv\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.198253 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.198218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fec38056-a2ad-4785-ac80-69dc61eb7d17-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.198253 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.198253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjnv\" (UniqueName: \"kubernetes.io/projected/fec38056-a2ad-4785-ac80-69dc61eb7d17-kube-api-access-vmjnv\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.198497 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.198287    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fec38056-a2ad-4785-ac80-69dc61eb7d17-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.198497 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.198335    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec38056-a2ad-4785-ac80-69dc61eb7d17-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.199018 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.198997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fec38056-a2ad-4785-ac80-69dc61eb7d17-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.200689 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.200671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fec38056-a2ad-4785-ac80-69dc61eb7d17-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.200777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.200758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fec38056-a2ad-4785-ac80-69dc61eb7d17-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.205861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.205836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjnv\" (UniqueName: \"kubernetes.io/projected/fec38056-a2ad-4785-ac80-69dc61eb7d17-kube-api-access-vmjnv\") pod \"prometheus-operator-6f4946db56-2c9fz\" (UID: \"fec38056-a2ad-4785-ac80-69dc61eb7d17\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.292839 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.292804    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"
Sep 09 17:21:39.570824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.570707    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2c9fz"]
Sep 09 17:21:39.573858 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:21:39.573825    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec38056_a2ad_4785_ac80_69dc61eb7d17.slice/crio-584cebfd53de36e6897af8e338af9fc51d369146dd98485079b693f8f2a7abb3 WatchSource:0}: Error finding container 584cebfd53de36e6897af8e338af9fc51d369146dd98485079b693f8f2a7abb3: Status 404 returned error can't find the container with id 584cebfd53de36e6897af8e338af9fc51d369146dd98485079b693f8f2a7abb3
Sep 09 17:21:39.825436 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:39.825399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz" event={"ID":"fec38056-a2ad-4785-ac80-69dc61eb7d17","Type":"ContainerStarted","Data":"584cebfd53de36e6897af8e338af9fc51d369146dd98485079b693f8f2a7abb3"}
Sep 09 17:21:41.836517 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:41.836484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz" event={"ID":"fec38056-a2ad-4785-ac80-69dc61eb7d17","Type":"ContainerStarted","Data":"d9beb5ee14d05dd892feb88f1db5ef76c48c0c827fa9e8a388e177d3d62a8157"}
Sep 09 17:21:41.838309 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:41.838275    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8b66885-x8xhk" event={"ID":"7e737cf2-0ee7-42b0-8529-ff441152ceed","Type":"ContainerStarted","Data":"ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843"}
Sep 09 17:21:41.855925 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:41.855875    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c8b66885-x8xhk" podStartSLOduration=1.480799672 podStartE2EDuration="4.855857737s" totalImagesPullingTime="3.375058065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:41.855058674 +0000 UTC m=+277.483211362" watchObservedRunningTime="2026-09-09 17:21:41.855857737 +0000 UTC m=+277.484010505"
Sep 09 17:21:42.843264 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:42.843231    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz" event={"ID":"fec38056-a2ad-4785-ac80-69dc61eb7d17","Type":"ContainerStarted","Data":"aafc1ea0e0b4208b899cb3cb873d6e8b77a15923ab4dc8d630dda23c8b8dfdb2"}
Sep 09 17:21:42.859122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:42.859074    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-2c9fz" podStartSLOduration=2.690715988 podStartE2EDuration="4.859058165s" totalImagesPullingTime="2.168342177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:42.8578889 +0000 UTC m=+278.486041604" watchObservedRunningTime="2026-09-09 17:21:42.859058165 +0000 UTC m=+278.487210846"
Sep 09 17:21:44.350712 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.350677    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5kp2k"]
Sep 09 17:21:44.353152 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.353132    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.355873 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.355854    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 17:21:44.356015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.355995    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4v4kg\""
Sep 09 17:21:44.356576 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.356560    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 17:21:44.356646 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.356580    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 17:21:44.435227 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435181    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-textfile\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435227 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435224    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435454 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-root\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435454 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1405646-f328-4dd5-8515-413e560f5748-metrics-client-ca\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435454 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q667h\" (UniqueName: \"kubernetes.io/projected/b1405646-f328-4dd5-8515-413e560f5748-kube-api-access-q667h\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-tls\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-sys\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.435568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.435547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-wtmp\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536500 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-textfile\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536500 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-root\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1405646-f328-4dd5-8515-413e560f5748-metrics-client-ca\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q667h\" (UniqueName: \"kubernetes.io/projected/b1405646-f328-4dd5-8515-413e560f5748-kube-api-access-q667h\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536612    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-root\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-tls\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-sys\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-wtmp\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-textfile\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.536943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-sys\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.537141 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.536988    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-wtmp\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.537256 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.537233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1405646-f328-4dd5-8515-413e560f5748-metrics-client-ca\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.537317 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.537290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.539021 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.538998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.539113 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.539045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1405646-f328-4dd5-8515-413e560f5748-node-exporter-tls\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.545528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.545506    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q667h\" (UniqueName: \"kubernetes.io/projected/b1405646-f328-4dd5-8515-413e560f5748-kube-api-access-q667h\") pod \"node-exporter-5kp2k\" (UID: \"b1405646-f328-4dd5-8515-413e560f5748\") " pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.662185 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.662099    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5kp2k"
Sep 09 17:21:44.672990 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:21:44.672938    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1405646_f328_4dd5_8515_413e560f5748.slice/crio-6271622c8ebf12b6f185b9fe146f9961e9461d04b19af29eccbcae1bf5367843 WatchSource:0}: Error finding container 6271622c8ebf12b6f185b9fe146f9961e9461d04b19af29eccbcae1bf5367843: Status 404 returned error can't find the container with id 6271622c8ebf12b6f185b9fe146f9961e9461d04b19af29eccbcae1bf5367843
Sep 09 17:21:44.851213 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:44.851176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kp2k" event={"ID":"b1405646-f328-4dd5-8515-413e560f5748","Type":"ContainerStarted","Data":"6271622c8ebf12b6f185b9fe146f9961e9461d04b19af29eccbcae1bf5367843"}
Sep 09 17:21:46.859683 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:46.859647    2558 generic.go:356] "Generic (PLEG): container finished" podID="b1405646-f328-4dd5-8515-413e560f5748" containerID="c8b5d096be735791cc98429bf62ca2e0d9e45cf37ff46bd7a56231cc94d13377" exitCode=0
Sep 09 17:21:46.860110 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:46.859704    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kp2k" event={"ID":"b1405646-f328-4dd5-8515-413e560f5748","Type":"ContainerDied","Data":"c8b5d096be735791cc98429bf62ca2e0d9e45cf37ff46bd7a56231cc94d13377"}
Sep 09 17:21:47.864859 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:47.864826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kp2k" event={"ID":"b1405646-f328-4dd5-8515-413e560f5748","Type":"ContainerStarted","Data":"c86efcd57dff09c6bcd1c83907795bc4a4f55339ccaed6e78eb4877bff06617e"}
Sep 09 17:21:47.865229 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:47.864875    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kp2k" event={"ID":"b1405646-f328-4dd5-8515-413e560f5748","Type":"ContainerStarted","Data":"5cafd4f628edc58bf84b1198aba056ab33a17aebf6eec35207b5b244a61cd138"}
Sep 09 17:21:47.884413 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:47.884362    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5kp2k" podStartSLOduration=2.381204978 podStartE2EDuration="3.884347571s" totalImagesPullingTime="1.437672628s" totalInitContainerRuntime="65.469965ms" isStatefulPod=true podCreationTimestamp="2026-09-09 17:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:47.883741532 +0000 UTC m=+283.511894236" watchObservedRunningTime="2026-09-09 17:21:47.884347571 +0000 UTC m=+283.512500287"
Sep 09 17:21:48.083185 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.083131    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:48.083383 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.083256    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:48.084614 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.084586    2558 patch_prober.go:28] interesting pod/console-5c8b66885-x8xhk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body=
Sep 09 17:21:48.084715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.084637    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5c8b66885-x8xhk" podUID="7e737cf2-0ee7-42b0-8529-ff441152ceed" containerName="console" probeResult="failure" output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused"
Sep 09 17:21:48.167857 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.167766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:21:48.170060 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.170027    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef37b751-5814-4867-930a-9632adbcc0c0-metrics-tls\") pod \"dns-default-hcclw\" (UID: \"ef37b751-5814-4867-930a-9632adbcc0c0\") " pod="openshift-dns/dns-default-hcclw"
Sep 09 17:21:48.215341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.215308    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cblwm\""
Sep 09 17:21:48.223550 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.223526    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hcclw"
Sep 09 17:21:48.490878 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.490844    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hcclw"]
Sep 09 17:21:48.493664 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:21:48.493639    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef37b751_5814_4867_930a_9632adbcc0c0.slice/crio-3659220c81fb56e5d69be2169bece0eb7367416c24aba4c73bcfde1c345ca4ec WatchSource:0}: Error finding container 3659220c81fb56e5d69be2169bece0eb7367416c24aba4c73bcfde1c345ca4ec: Status 404 returned error can't find the container with id 3659220c81fb56e5d69be2169bece0eb7367416c24aba4c73bcfde1c345ca4ec
Sep 09 17:21:48.869257 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:48.869177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hcclw" event={"ID":"ef37b751-5814-4867-930a-9632adbcc0c0","Type":"ContainerStarted","Data":"3659220c81fb56e5d69be2169bece0eb7367416c24aba4c73bcfde1c345ca4ec"}
Sep 09 17:21:50.879118 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:50.879067    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hcclw" event={"ID":"ef37b751-5814-4867-930a-9632adbcc0c0","Type":"ContainerStarted","Data":"15fa8fcd709a471d1f625ea9776a98fdb26b11ad04f6339eb1307f26562e2df0"}
Sep 09 17:21:50.879510 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:50.879136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hcclw" event={"ID":"ef37b751-5814-4867-930a-9632adbcc0c0","Type":"ContainerStarted","Data":"9004621489b1e4f16dbf676bcf640e091693d0d1c79d656500dc0be65ed54b21"}
Sep 09 17:21:50.879510 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:50.879182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hcclw"
Sep 09 17:21:50.898521 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:50.898467    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hcclw" podStartSLOduration=251.120094965 podStartE2EDuration="4m12.898449201s" totalImagesPullingTime="1.778354236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:50.897638537 +0000 UTC m=+286.525791242" watchObservedRunningTime="2026-09-09 17:21:50.898449201 +0000 UTC m=+286.526601902"
Sep 09 17:21:58.087763 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:58.087728    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:21:58.091556 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:21:58.091538    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:22:00.884509 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:00.884478    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hcclw"
Sep 09 17:22:01.201335 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:01.201300    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c8b66885-x8xhk"]
Sep 09 17:22:04.697862 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:04.697832    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:22:04.698312 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:04.697946    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:22:04.700533 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:04.700512    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 17:22:17.849699 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:17.849628    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" podUID="319267dd-5f0d-42db-a56c-327cd573c9fb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 17:22:26.220830 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.220752    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c8b66885-x8xhk" podUID="7e737cf2-0ee7-42b0-8529-ff441152ceed" containerName="console" containerID="cri-o://ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843" gracePeriod=15
Sep 09 17:22:26.463602 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.463575    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:22:26.536427 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536335    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx9lb\" (UniqueName: \"kubernetes.io/projected/7e737cf2-0ee7-42b0-8529-ff441152ceed-kube-api-access-hx9lb\") pod \"7e737cf2-0ee7-42b0-8529-ff441152ceed\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") "
Sep 09 17:22:26.536427 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536372    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-service-ca\") pod \"7e737cf2-0ee7-42b0-8529-ff441152ceed\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") "
Sep 09 17:22:26.536427 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536404    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-oauth-serving-cert\") pod \"7e737cf2-0ee7-42b0-8529-ff441152ceed\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") "
Sep 09 17:22:26.536711 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536440    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-oauth-config\") pod \"7e737cf2-0ee7-42b0-8529-ff441152ceed\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") "
Sep 09 17:22:26.536711 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536472    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-config\") pod \"7e737cf2-0ee7-42b0-8529-ff441152ceed\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") "
Sep 09 17:22:26.536711 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536492    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-serving-cert\") pod \"7e737cf2-0ee7-42b0-8529-ff441152ceed\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") "
Sep 09 17:22:26.536711 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536513    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-trusted-ca-bundle\") pod \"7e737cf2-0ee7-42b0-8529-ff441152ceed\" (UID: \"7e737cf2-0ee7-42b0-8529-ff441152ceed\") "
Sep 09 17:22:26.536908 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536853    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7e737cf2-0ee7-42b0-8529-ff441152ceed" (UID: "7e737cf2-0ee7-42b0-8529-ff441152ceed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:22:26.537014 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.536908    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-config" (OuterVolumeSpecName: "console-config") pod "7e737cf2-0ee7-42b0-8529-ff441152ceed" (UID: "7e737cf2-0ee7-42b0-8529-ff441152ceed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:22:26.537118 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.537024    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7e737cf2-0ee7-42b0-8529-ff441152ceed" (UID: "7e737cf2-0ee7-42b0-8529-ff441152ceed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:22:26.537169 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.537107    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-service-ca" (OuterVolumeSpecName: "service-ca") pod "7e737cf2-0ee7-42b0-8529-ff441152ceed" (UID: "7e737cf2-0ee7-42b0-8529-ff441152ceed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:22:26.538754 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.538725    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7e737cf2-0ee7-42b0-8529-ff441152ceed" (UID: "7e737cf2-0ee7-42b0-8529-ff441152ceed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:22:26.538822 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.538765    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7e737cf2-0ee7-42b0-8529-ff441152ceed" (UID: "7e737cf2-0ee7-42b0-8529-ff441152ceed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:22:26.538857 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.538814    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e737cf2-0ee7-42b0-8529-ff441152ceed-kube-api-access-hx9lb" (OuterVolumeSpecName: "kube-api-access-hx9lb") pod "7e737cf2-0ee7-42b0-8529-ff441152ceed" (UID: "7e737cf2-0ee7-42b0-8529-ff441152ceed"). InnerVolumeSpecName "kube-api-access-hx9lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:22:26.637211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.637173    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-trusted-ca-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:22:26.637211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.637201    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9lb\" (UniqueName: \"kubernetes.io/projected/7e737cf2-0ee7-42b0-8529-ff441152ceed-kube-api-access-hx9lb\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:22:26.637211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.637212    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:22:26.637211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.637221    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-oauth-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:22:26.637465 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.637229    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-oauth-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:22:26.637465 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.637238    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-config\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:22:26.637465 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.637246    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e737cf2-0ee7-42b0-8529-ff441152ceed-console-serving-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:22:26.999580 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.999547    2558 generic.go:356] "Generic (PLEG): container finished" podID="7e737cf2-0ee7-42b0-8529-ff441152ceed" containerID="ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843" exitCode=0
Sep 09 17:22:26.999745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.999594    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c8b66885-x8xhk"
Sep 09 17:22:26.999745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.999600    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8b66885-x8xhk" event={"ID":"7e737cf2-0ee7-42b0-8529-ff441152ceed","Type":"ContainerDied","Data":"ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843"}
Sep 09 17:22:26.999745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.999639    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c8b66885-x8xhk" event={"ID":"7e737cf2-0ee7-42b0-8529-ff441152ceed","Type":"ContainerDied","Data":"c0a5bb743593682cb916dfbbb90b07601cf69bb302e16ad5008b9db98d32d696"}
Sep 09 17:22:26.999745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:26.999670    2558 scope.go:117] "RemoveContainer" containerID="ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843"
Sep 09 17:22:27.007858 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:27.007836    2558 scope.go:117] "RemoveContainer" containerID="ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843"
Sep 09 17:22:27.008193 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:22:27.008172    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843\": container with ID starting with ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843 not found: ID does not exist" containerID="ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843"
Sep 09 17:22:27.008274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:27.008206    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843"} err="failed to get container status \"ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843\": rpc error: code = NotFound desc = could not find container \"ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843\": container with ID starting with ff82d39009dd95312200c186814ca7d1c25a58c565b6a2ee152c1b98e0781843 not found: ID does not exist"
Sep 09 17:22:27.015723 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:27.015693    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c8b66885-x8xhk"]
Sep 09 17:22:27.017624 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:27.017599    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c8b66885-x8xhk"]
Sep 09 17:22:27.302703 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:27.302677    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qpr8j_b07f3322-92ad-4575-af6a-ae77de513515/node-ca/0.log"
Sep 09 17:22:27.849999 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:27.849943    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" podUID="319267dd-5f0d-42db-a56c-327cd573c9fb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 17:22:28.499949 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:28.499914    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qdcbd_c13bf76c-fc53-4e46-a5ce-4a4b5e37c867/serve-healthcheck-canary/0.log"
Sep 09 17:22:28.815162 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:28.815084    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e737cf2-0ee7-42b0-8529-ff441152ceed" path="/var/lib/kubelet/pods/7e737cf2-0ee7-42b0-8529-ff441152ceed/volumes"
Sep 09 17:22:37.849741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:37.849685    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" podUID="319267dd-5f0d-42db-a56c-327cd573c9fb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 17:22:37.850185 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:37.849781    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb"
Sep 09 17:22:37.850330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:37.850310    2558 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"20a104161e4852170ff7012aa30600020ede503b1cbbdec969d73c200bda7b97"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 17:22:37.850376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:37.850353    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" podUID="319267dd-5f0d-42db-a56c-327cd573c9fb" containerName="service-proxy" containerID="cri-o://20a104161e4852170ff7012aa30600020ede503b1cbbdec969d73c200bda7b97" gracePeriod=30
Sep 09 17:22:38.043161 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:38.043103    2558 generic.go:356] "Generic (PLEG): container finished" podID="319267dd-5f0d-42db-a56c-327cd573c9fb" containerID="20a104161e4852170ff7012aa30600020ede503b1cbbdec969d73c200bda7b97" exitCode=2
Sep 09 17:22:38.043161 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:38.043163    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" event={"ID":"319267dd-5f0d-42db-a56c-327cd573c9fb","Type":"ContainerDied","Data":"20a104161e4852170ff7012aa30600020ede503b1cbbdec969d73c200bda7b97"}
Sep 09 17:22:38.043410 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:22:38.043209    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59d87c9599-69nqb" event={"ID":"319267dd-5f0d-42db-a56c-327cd573c9fb","Type":"ContainerStarted","Data":"795760e68c17f794a8c4adc3eaca22c8de2edc7ef466f3c008812cebec88698f"}
Sep 09 17:25:34.660058 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.660022    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"]
Sep 09 17:25:34.660475 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.660250    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e737cf2-0ee7-42b0-8529-ff441152ceed" containerName="console"
Sep 09 17:25:34.660475 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.660262    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e737cf2-0ee7-42b0-8529-ff441152ceed" containerName="console"
Sep 09 17:25:34.660475 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.660304    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e737cf2-0ee7-42b0-8529-ff441152ceed" containerName="console"
Sep 09 17:25:34.663040 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.663018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.665687 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.665662    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 17:25:34.666764 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.666747    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rw5z\""
Sep 09 17:25:34.666848 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.666804    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 17:25:34.671669 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.671644    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"]
Sep 09 17:25:34.727794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.727754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.727991 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.727810    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdzx\" (UniqueName: \"kubernetes.io/projected/ed4d901c-6c53-48ed-b70b-cd3520727c4d-kube-api-access-jzdzx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.727991 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.727874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.828237 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.828207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.828418 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.828282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.828418 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.828314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdzx\" (UniqueName: \"kubernetes.io/projected/ed4d901c-6c53-48ed-b70b-cd3520727c4d-kube-api-access-jzdzx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.828721 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.828685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.828721 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.828705    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.837214 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.837180    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdzx\" (UniqueName: \"kubernetes.io/projected/ed4d901c-6c53-48ed-b70b-cd3520727c4d-kube-api-access-jzdzx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:34.971861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:34.971819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:35.239948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:35.239868    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"]
Sep 09 17:25:35.244585 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:25:35.244557    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4d901c_6c53_48ed_b70b_cd3520727c4d.slice/crio-adc1a28c0fee844d1c124a994c55d12bb7fe92d39c3df812c0c357ef5d83796d WatchSource:0}: Error finding container adc1a28c0fee844d1c124a994c55d12bb7fe92d39c3df812c0c357ef5d83796d: Status 404 returned error can't find the container with id adc1a28c0fee844d1c124a994c55d12bb7fe92d39c3df812c0c357ef5d83796d
Sep 09 17:25:35.246563 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:35.246542    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 17:25:35.616130 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:35.616041    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp" event={"ID":"ed4d901c-6c53-48ed-b70b-cd3520727c4d","Type":"ContainerStarted","Data":"adc1a28c0fee844d1c124a994c55d12bb7fe92d39c3df812c0c357ef5d83796d"}
Sep 09 17:25:43.649188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:43.649153    2558 generic.go:356] "Generic (PLEG): container finished" podID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerID="49a248ef58d65738602637bef160ea2706b425ef4cad6e942e4288365a035fe0" exitCode=0
Sep 09 17:25:43.649596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:43.649230    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp" event={"ID":"ed4d901c-6c53-48ed-b70b-cd3520727c4d","Type":"ContainerDied","Data":"49a248ef58d65738602637bef160ea2706b425ef4cad6e942e4288365a035fe0"}
Sep 09 17:25:46.662512 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:46.662475    2558 generic.go:356] "Generic (PLEG): container finished" podID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerID="e08f34af741dc4d64c35f9d233bb3644d4efda7af79077f9424d8052c0164900" exitCode=0
Sep 09 17:25:46.662879 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:46.662570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp" event={"ID":"ed4d901c-6c53-48ed-b70b-cd3520727c4d","Type":"ContainerDied","Data":"e08f34af741dc4d64c35f9d233bb3644d4efda7af79077f9424d8052c0164900"}
Sep 09 17:25:53.690919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:53.690884    2558 generic.go:356] "Generic (PLEG): container finished" podID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerID="58a659242e3f20f3c9f1aa524a880d4012e24382b01fddc43ccde03105ccd55c" exitCode=0
Sep 09 17:25:53.691337 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:53.690984    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp" event={"ID":"ed4d901c-6c53-48ed-b70b-cd3520727c4d","Type":"ContainerDied","Data":"58a659242e3f20f3c9f1aa524a880d4012e24382b01fddc43ccde03105ccd55c"}
Sep 09 17:25:54.825400 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:54.825371    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:54.982211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:54.982124    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-util\") pod \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") "
Sep 09 17:25:54.982211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:54.982183    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-bundle\") pod \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") "
Sep 09 17:25:54.982388 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:54.982249    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzdzx\" (UniqueName: \"kubernetes.io/projected/ed4d901c-6c53-48ed-b70b-cd3520727c4d-kube-api-access-jzdzx\") pod \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\" (UID: \"ed4d901c-6c53-48ed-b70b-cd3520727c4d\") "
Sep 09 17:25:54.982902 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:54.982868    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-bundle" (OuterVolumeSpecName: "bundle") pod "ed4d901c-6c53-48ed-b70b-cd3520727c4d" (UID: "ed4d901c-6c53-48ed-b70b-cd3520727c4d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:25:54.984477 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:54.984446    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4d901c-6c53-48ed-b70b-cd3520727c4d-kube-api-access-jzdzx" (OuterVolumeSpecName: "kube-api-access-jzdzx") pod "ed4d901c-6c53-48ed-b70b-cd3520727c4d" (UID: "ed4d901c-6c53-48ed-b70b-cd3520727c4d"). InnerVolumeSpecName "kube-api-access-jzdzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:25:54.987159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:54.987129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-util" (OuterVolumeSpecName: "util") pod "ed4d901c-6c53-48ed-b70b-cd3520727c4d" (UID: "ed4d901c-6c53-48ed-b70b-cd3520727c4d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:25:55.082847 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:55.082802    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdzx\" (UniqueName: \"kubernetes.io/projected/ed4d901c-6c53-48ed-b70b-cd3520727c4d-kube-api-access-jzdzx\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:25:55.082847 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:55.082839    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:25:55.082847 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:55.082850    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed4d901c-6c53-48ed-b70b-cd3520727c4d-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:25:55.699677 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:55.699642    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp" event={"ID":"ed4d901c-6c53-48ed-b70b-cd3520727c4d","Type":"ContainerDied","Data":"adc1a28c0fee844d1c124a994c55d12bb7fe92d39c3df812c0c357ef5d83796d"}
Sep 09 17:25:55.699835 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:55.699685    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sdxrp"
Sep 09 17:25:55.699835 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:25:55.699718    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc1a28c0fee844d1c124a994c55d12bb7fe92d39c3df812c0c357ef5d83796d"
Sep 09 17:26:23.131646 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131607    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"]
Sep 09 17:26:23.132142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131845    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerName="pull"
Sep 09 17:26:23.132142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131860    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerName="pull"
Sep 09 17:26:23.132142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131877    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerName="extract"
Sep 09 17:26:23.132142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131883    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerName="extract"
Sep 09 17:26:23.132142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131895    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerName="util"
Sep 09 17:26:23.132142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131901    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerName="util"
Sep 09 17:26:23.132142 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.131942    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4d901c-6c53-48ed-b70b-cd3520727c4d" containerName="extract"
Sep 09 17:26:23.134094 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.134077    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.137435 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.137411    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 17:26:23.137572 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.137476    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 17:26:23.138434 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.138417    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rw5z\""
Sep 09 17:26:23.142305 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.142265    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"]
Sep 09 17:26:23.164974 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.164925    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59wzk\" (UniqueName: \"kubernetes.io/projected/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-kube-api-access-59wzk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.165130 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.164985    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.165130 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.165045    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.265391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.265348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59wzk\" (UniqueName: \"kubernetes.io/projected/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-kube-api-access-59wzk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.265579 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.265400    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.265579 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.265438    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.265801 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.265780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.265862 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.265807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.273974 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.273923    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59wzk\" (UniqueName: \"kubernetes.io/projected/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-kube-api-access-59wzk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.443387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.443359    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:23.709850 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.709825    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"]
Sep 09 17:26:23.712178 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:26:23.712151    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f39cf55_f7b6_4857_9f20_f07fc8faf11f.slice/crio-f5f710d9a5da4ccd6b4cb95a10538e2864712fa383c2b0e2fb9dafdae0c0152e WatchSource:0}: Error finding container f5f710d9a5da4ccd6b4cb95a10538e2864712fa383c2b0e2fb9dafdae0c0152e: Status 404 returned error can't find the container with id f5f710d9a5da4ccd6b4cb95a10538e2864712fa383c2b0e2fb9dafdae0c0152e
Sep 09 17:26:23.796023 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.795988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv" event={"ID":"9f39cf55-f7b6-4857-9f20-f07fc8faf11f","Type":"ContainerStarted","Data":"0ee3404ee8c6e4c83be84601ab784fff82e6e6c8e29edc0b5b0aa3f6e9118e94"}
Sep 09 17:26:23.796175 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:23.796040    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv" event={"ID":"9f39cf55-f7b6-4857-9f20-f07fc8faf11f","Type":"ContainerStarted","Data":"f5f710d9a5da4ccd6b4cb95a10538e2864712fa383c2b0e2fb9dafdae0c0152e"}
Sep 09 17:26:24.801505 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:24.801469    2558 generic.go:356] "Generic (PLEG): container finished" podID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerID="0ee3404ee8c6e4c83be84601ab784fff82e6e6c8e29edc0b5b0aa3f6e9118e94" exitCode=0
Sep 09 17:26:24.801505 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:24.801508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv" event={"ID":"9f39cf55-f7b6-4857-9f20-f07fc8faf11f","Type":"ContainerDied","Data":"0ee3404ee8c6e4c83be84601ab784fff82e6e6c8e29edc0b5b0aa3f6e9118e94"}
Sep 09 17:26:28.816564 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:28.816524    2558 generic.go:356] "Generic (PLEG): container finished" podID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerID="01246dfc67da7a379f1ea57a5d0fea4c70edf4ad0d6e591510647f932af9c260" exitCode=0
Sep 09 17:26:28.817086 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:28.816605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv" event={"ID":"9f39cf55-f7b6-4857-9f20-f07fc8faf11f","Type":"ContainerDied","Data":"01246dfc67da7a379f1ea57a5d0fea4c70edf4ad0d6e591510647f932af9c260"}
Sep 09 17:26:29.822450 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:29.822414    2558 generic.go:356] "Generic (PLEG): container finished" podID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerID="d1593b7af20b48e7e3f9591e3034b3bc21a8ea2e060737d583140afc5d0cc47f" exitCode=0
Sep 09 17:26:29.822818 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:29.822505    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv" event={"ID":"9f39cf55-f7b6-4857-9f20-f07fc8faf11f","Type":"ContainerDied","Data":"d1593b7af20b48e7e3f9591e3034b3bc21a8ea2e060737d583140afc5d0cc47f"}
Sep 09 17:26:30.942682 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:30.942652    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:31.019070 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.019028    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59wzk\" (UniqueName: \"kubernetes.io/projected/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-kube-api-access-59wzk\") pod \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") "
Sep 09 17:26:31.019253 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.019103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-bundle\") pod \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") "
Sep 09 17:26:31.019253 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.019126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-util\") pod \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\" (UID: \"9f39cf55-f7b6-4857-9f20-f07fc8faf11f\") "
Sep 09 17:26:31.019539 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.019511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-bundle" (OuterVolumeSpecName: "bundle") pod "9f39cf55-f7b6-4857-9f20-f07fc8faf11f" (UID: "9f39cf55-f7b6-4857-9f20-f07fc8faf11f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:26:31.021195 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.021171    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-kube-api-access-59wzk" (OuterVolumeSpecName: "kube-api-access-59wzk") pod "9f39cf55-f7b6-4857-9f20-f07fc8faf11f" (UID: "9f39cf55-f7b6-4857-9f20-f07fc8faf11f"). InnerVolumeSpecName "kube-api-access-59wzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:26:31.026106 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.026081    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-util" (OuterVolumeSpecName: "util") pod "9f39cf55-f7b6-4857-9f20-f07fc8faf11f" (UID: "9f39cf55-f7b6-4857-9f20-f07fc8faf11f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:26:31.120176 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.120078    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:26:31.120176 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.120115    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:26:31.120176 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.120125    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59wzk\" (UniqueName: \"kubernetes.io/projected/9f39cf55-f7b6-4857-9f20-f07fc8faf11f-kube-api-access-59wzk\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:26:31.830691 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.830650    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv" event={"ID":"9f39cf55-f7b6-4857-9f20-f07fc8faf11f","Type":"ContainerDied","Data":"f5f710d9a5da4ccd6b4cb95a10538e2864712fa383c2b0e2fb9dafdae0c0152e"}
Sep 09 17:26:31.830691 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.830685    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5xxwv"
Sep 09 17:26:31.830887 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:31.830709    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5f710d9a5da4ccd6b4cb95a10538e2864712fa383c2b0e2fb9dafdae0c0152e"
Sep 09 17:26:35.373406 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373377    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"]
Sep 09 17:26:35.373777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373588    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerName="util"
Sep 09 17:26:35.373777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373599    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerName="util"
Sep 09 17:26:35.373777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373611    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerName="pull"
Sep 09 17:26:35.373777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373616    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerName="pull"
Sep 09 17:26:35.373777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373627    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerName="extract"
Sep 09 17:26:35.373777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373634    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerName="extract"
Sep 09 17:26:35.373777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.373671    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f39cf55-f7b6-4857-9f20-f07fc8faf11f" containerName="extract"
Sep 09 17:26:35.375428 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.375412    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.377978 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.377935    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 17:26:35.379323 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.379305    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7rp97\""
Sep 09 17:26:35.379323 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.379317    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 17:26:35.384548 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.384527    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"]
Sep 09 17:26:35.450266 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.450225    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zswbz\" (UID: \"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.450440 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.450283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9vs\" (UniqueName: \"kubernetes.io/projected/c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18-kube-api-access-xn9vs\") pod \"openshift-lws-operator-fd8ccff4c-zswbz\" (UID: \"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.551577 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.551540    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zswbz\" (UID: \"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.551712 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.551608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9vs\" (UniqueName: \"kubernetes.io/projected/c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18-kube-api-access-xn9vs\") pod \"openshift-lws-operator-fd8ccff4c-zswbz\" (UID: \"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.551999 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.551942    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zswbz\" (UID: \"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.559747 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.559726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9vs\" (UniqueName: \"kubernetes.io/projected/c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18-kube-api-access-xn9vs\") pod \"openshift-lws-operator-fd8ccff4c-zswbz\" (UID: \"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.684518 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.684487    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"
Sep 09 17:26:35.956541 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:35.956394    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz"]
Sep 09 17:26:35.959239 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:26:35.959208    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc21d2dfd_0f4b_4cc0_82fa_67a5f107ba18.slice/crio-7c798601385d44b8a8d6ecbe8593fa95b424c9002d5aaefe97592bbcc4a33aca WatchSource:0}: Error finding container 7c798601385d44b8a8d6ecbe8593fa95b424c9002d5aaefe97592bbcc4a33aca: Status 404 returned error can't find the container with id 7c798601385d44b8a8d6ecbe8593fa95b424c9002d5aaefe97592bbcc4a33aca
Sep 09 17:26:36.850715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:36.850676    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz" event={"ID":"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18","Type":"ContainerStarted","Data":"7c798601385d44b8a8d6ecbe8593fa95b424c9002d5aaefe97592bbcc4a33aca"}
Sep 09 17:26:37.856151 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:37.856116    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz" event={"ID":"c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18","Type":"ContainerStarted","Data":"b1ab428d1470023c19f65b8a019d1dd483f3e1b8b7685b06365424b7b61cc343"}
Sep 09 17:26:37.872337 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:37.872282    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zswbz" podStartSLOduration=1.192208517 podStartE2EDuration="2.872266998s" totalImagesPullingTime="1.680058481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:26:37.871703069 +0000 UTC m=+573.499855772" watchObservedRunningTime="2026-09-09 17:26:37.872266998 +0000 UTC m=+573.500419701"
Sep 09 17:26:38.028494 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.028460    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65f65c7f66-hfmc9"]
Sep 09 17:26:38.030461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.030446    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.033388 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033301    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 17:26:38.033388 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033320    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 17:26:38.033388 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033357    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 17:26:38.033628 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033360    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 17:26:38.033628 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033590    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 17:26:38.033728 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 17:26:38.033780 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033768    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rfpcp\""
Sep 09 17:26:38.033869 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.033850    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 17:26:38.039176 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.039156    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 17:26:38.043444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.043421    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f65c7f66-hfmc9"]
Sep 09 17:26:38.072698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.072669    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-serving-cert\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.072698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.072696    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-config\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.072919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.072713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-trusted-ca-bundle\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.072919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.072739    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-oauth-config\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.072919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.072757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-oauth-serving-cert\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.072919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.072822    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-service-ca\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.072919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.072858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntm8\" (UniqueName: \"kubernetes.io/projected/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-kube-api-access-cntm8\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.174185 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.174144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-oauth-config\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.174185 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.174190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-oauth-serving-cert\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.174373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.174226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-service-ca\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.174373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.174263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntm8\" (UniqueName: \"kubernetes.io/projected/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-kube-api-access-cntm8\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.174373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.174317    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-serving-cert\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.174373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.174339    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-config\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.174373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.174362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-trusted-ca-bundle\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.175235 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.175206    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-service-ca\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.175361 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.175245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-config\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.175361 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.175314    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-oauth-serving-cert\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.175361 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.175354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-trusted-ca-bundle\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.176663 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.176633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-oauth-config\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.176796 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.176780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-console-serving-cert\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.183696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.183672    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntm8\" (UniqueName: \"kubernetes.io/projected/27d7169e-87bf-47cb-abc2-ea4bc55c4ff7-kube-api-access-cntm8\") pod \"console-65f65c7f66-hfmc9\" (UID: \"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7\") " pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.340360 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.340283    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:38.610799 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.610771    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f65c7f66-hfmc9"]
Sep 09 17:26:38.613023 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:26:38.612992    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d7169e_87bf_47cb_abc2_ea4bc55c4ff7.slice/crio-7da9b39b3e983cef8081e05c8cfade66fe6a0e7b8548a76341c0b3770738fa59 WatchSource:0}: Error finding container 7da9b39b3e983cef8081e05c8cfade66fe6a0e7b8548a76341c0b3770738fa59: Status 404 returned error can't find the container with id 7da9b39b3e983cef8081e05c8cfade66fe6a0e7b8548a76341c0b3770738fa59
Sep 09 17:26:38.861642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.861541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f65c7f66-hfmc9" event={"ID":"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7","Type":"ContainerStarted","Data":"cffebffb1accb0cf5041aaf3eabdf692c1f3b34e212c1f00fd3858f690c3207d"}
Sep 09 17:26:38.861642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.861596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f65c7f66-hfmc9" event={"ID":"27d7169e-87bf-47cb-abc2-ea4bc55c4ff7","Type":"ContainerStarted","Data":"7da9b39b3e983cef8081e05c8cfade66fe6a0e7b8548a76341c0b3770738fa59"}
Sep 09 17:26:38.878186 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:38.878135    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65f65c7f66-hfmc9" podStartSLOduration=0.878120416 podStartE2EDuration="878.120416ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:26:38.877765748 +0000 UTC m=+574.505918450" watchObservedRunningTime="2026-09-09 17:26:38.878120416 +0000 UTC m=+574.506273118"
Sep 09 17:26:40.767371 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.767332    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"]
Sep 09 17:26:40.769500 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.769483    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.772224 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.772204    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 17:26:40.773677 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.773658    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 17:26:40.773778 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.773672    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rw5z\""
Sep 09 17:26:40.778141 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.778122    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"]
Sep 09 17:26:40.793060 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.793035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.793184 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.793065    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.793184 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.793089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbkn\" (UniqueName: \"kubernetes.io/projected/edf6073f-2996-4998-aa3b-d4c927409735-kube-api-access-lzbkn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.894261 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.894226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.894261 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.894265    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbkn\" (UniqueName: \"kubernetes.io/projected/edf6073f-2996-4998-aa3b-d4c927409735-kube-api-access-lzbkn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.894513 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.894341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.894763 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.894738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.894824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.894748    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:40.903108 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:40.903062    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbkn\" (UniqueName: \"kubernetes.io/projected/edf6073f-2996-4998-aa3b-d4c927409735-kube-api-access-lzbkn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:41.078745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:41.078662    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:41.349191 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:41.349117    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"]
Sep 09 17:26:41.352602 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:26:41.352572    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf6073f_2996_4998_aa3b_d4c927409735.slice/crio-a13d75b1b29226a24c0719020675848218ed0836823dc98fa15c666f8407449f WatchSource:0}: Error finding container a13d75b1b29226a24c0719020675848218ed0836823dc98fa15c666f8407449f: Status 404 returned error can't find the container with id a13d75b1b29226a24c0719020675848218ed0836823dc98fa15c666f8407449f
Sep 09 17:26:41.874787 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:41.874754    2558 generic.go:356] "Generic (PLEG): container finished" podID="edf6073f-2996-4998-aa3b-d4c927409735" containerID="b0c876958b5f310f6dec9f57d16d1127c4844725557bd05b78f1e58bbc71c0c0" exitCode=0
Sep 09 17:26:41.875267 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:41.874825    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4" event={"ID":"edf6073f-2996-4998-aa3b-d4c927409735","Type":"ContainerDied","Data":"b0c876958b5f310f6dec9f57d16d1127c4844725557bd05b78f1e58bbc71c0c0"}
Sep 09 17:26:41.875267 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:41.874856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4" event={"ID":"edf6073f-2996-4998-aa3b-d4c927409735","Type":"ContainerStarted","Data":"a13d75b1b29226a24c0719020675848218ed0836823dc98fa15c666f8407449f"}
Sep 09 17:26:42.880635 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:42.880546    2558 generic.go:356] "Generic (PLEG): container finished" podID="edf6073f-2996-4998-aa3b-d4c927409735" containerID="f66adb26834e0db5820e7b61dea64c465eb29e98103e7151581a5137995c2b12" exitCode=0
Sep 09 17:26:42.880635 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:42.880611    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4" event={"ID":"edf6073f-2996-4998-aa3b-d4c927409735","Type":"ContainerDied","Data":"f66adb26834e0db5820e7b61dea64c465eb29e98103e7151581a5137995c2b12"}
Sep 09 17:26:43.886635 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:43.886602    2558 generic.go:356] "Generic (PLEG): container finished" podID="edf6073f-2996-4998-aa3b-d4c927409735" containerID="598a4010666313c94b367b7f4e301d4518b51523b0ff77749fbe92545df8c6ec" exitCode=0
Sep 09 17:26:43.887014 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:43.886660    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4" event={"ID":"edf6073f-2996-4998-aa3b-d4c927409735","Type":"ContainerDied","Data":"598a4010666313c94b367b7f4e301d4518b51523b0ff77749fbe92545df8c6ec"}
Sep 09 17:26:45.007930 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.007905    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:45.124508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.124466    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbkn\" (UniqueName: \"kubernetes.io/projected/edf6073f-2996-4998-aa3b-d4c927409735-kube-api-access-lzbkn\") pod \"edf6073f-2996-4998-aa3b-d4c927409735\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") "
Sep 09 17:26:45.124669 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.124545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-util\") pod \"edf6073f-2996-4998-aa3b-d4c927409735\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") "
Sep 09 17:26:45.124669 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.124570    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-bundle\") pod \"edf6073f-2996-4998-aa3b-d4c927409735\" (UID: \"edf6073f-2996-4998-aa3b-d4c927409735\") "
Sep 09 17:26:45.125351 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.125322    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-bundle" (OuterVolumeSpecName: "bundle") pod "edf6073f-2996-4998-aa3b-d4c927409735" (UID: "edf6073f-2996-4998-aa3b-d4c927409735"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:26:45.126549 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.126524    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf6073f-2996-4998-aa3b-d4c927409735-kube-api-access-lzbkn" (OuterVolumeSpecName: "kube-api-access-lzbkn") pod "edf6073f-2996-4998-aa3b-d4c927409735" (UID: "edf6073f-2996-4998-aa3b-d4c927409735"). InnerVolumeSpecName "kube-api-access-lzbkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:26:45.129286 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.129265    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-util" (OuterVolumeSpecName: "util") pod "edf6073f-2996-4998-aa3b-d4c927409735" (UID: "edf6073f-2996-4998-aa3b-d4c927409735"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:26:45.225269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.225230    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbkn\" (UniqueName: \"kubernetes.io/projected/edf6073f-2996-4998-aa3b-d4c927409735-kube-api-access-lzbkn\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:26:45.225269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.225263    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:26:45.225444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.225296    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6073f-2996-4998-aa3b-d4c927409735-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:26:45.895790 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.895757    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4"
Sep 09 17:26:45.895969 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.895760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9clj4" event={"ID":"edf6073f-2996-4998-aa3b-d4c927409735","Type":"ContainerDied","Data":"a13d75b1b29226a24c0719020675848218ed0836823dc98fa15c666f8407449f"}
Sep 09 17:26:45.895969 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:45.895878    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a13d75b1b29226a24c0719020675848218ed0836823dc98fa15c666f8407449f"
Sep 09 17:26:48.340862 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:48.340824    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:48.341258 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:48.340879    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:48.345496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:48.345474    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:48.912595 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:48.912568    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65f65c7f66-hfmc9"
Sep 09 17:26:53.732176 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732142    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"]
Sep 09 17:26:53.732581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732420    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6073f-2996-4998-aa3b-d4c927409735" containerName="extract"
Sep 09 17:26:53.732581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732438    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6073f-2996-4998-aa3b-d4c927409735" containerName="extract"
Sep 09 17:26:53.732581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732450    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6073f-2996-4998-aa3b-d4c927409735" containerName="util"
Sep 09 17:26:53.732581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732455    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6073f-2996-4998-aa3b-d4c927409735" containerName="util"
Sep 09 17:26:53.732581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732466    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6073f-2996-4998-aa3b-d4c927409735" containerName="pull"
Sep 09 17:26:53.732581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732474    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6073f-2996-4998-aa3b-d4c927409735" containerName="pull"
Sep 09 17:26:53.732581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.732519    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf6073f-2996-4998-aa3b-d4c927409735" containerName="extract"
Sep 09 17:26:53.734601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.734574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.738896 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.738866    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 17:26:53.738896 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.738866    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vq5nr\""
Sep 09 17:26:53.739121 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.738902    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 17:26:53.739282 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.739263    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 17:26:53.748787 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.748763    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"]
Sep 09 17:26:53.779320 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.779291    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-manager-config\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.779320 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.779323    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lxp8\" (UniqueName: \"kubernetes.io/projected/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-kube-api-access-6lxp8\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.779522 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.779344    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-metrics-cert\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.779522 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.779426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-cert\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.880737 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.880693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-manager-config\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.880737 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.880737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lxp8\" (UniqueName: \"kubernetes.io/projected/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-kube-api-access-6lxp8\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.880927 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.880758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-metrics-cert\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.880927 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.880795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-cert\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.881497 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.881458    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-manager-config\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.883601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.883571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-cert\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.883790 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.883754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-metrics-cert\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:53.889797 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:53.889768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lxp8\" (UniqueName: \"kubernetes.io/projected/b01bb6bd-0677-4010-9acd-3aa55a7d75dc-kube-api-access-6lxp8\") pod \"lws-controller-manager-67d8946886-rmz2c\" (UID: \"b01bb6bd-0677-4010-9acd-3aa55a7d75dc\") " pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:54.044341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:54.044234    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:26:54.314037 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:54.313904    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"]
Sep 09 17:26:54.316019 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:26:54.315986    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01bb6bd_0677_4010_9acd_3aa55a7d75dc.slice/crio-f38bdfe8ab1b1be4a495cc2807e8173d08d30a712543b61b9a9951996175ff7e WatchSource:0}: Error finding container f38bdfe8ab1b1be4a495cc2807e8173d08d30a712543b61b9a9951996175ff7e: Status 404 returned error can't find the container with id f38bdfe8ab1b1be4a495cc2807e8173d08d30a712543b61b9a9951996175ff7e
Sep 09 17:26:54.930326 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:54.930291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c" event={"ID":"b01bb6bd-0677-4010-9acd-3aa55a7d75dc","Type":"ContainerStarted","Data":"f38bdfe8ab1b1be4a495cc2807e8173d08d30a712543b61b9a9951996175ff7e"}
Sep 09 17:26:57.261919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.261879    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"]
Sep 09 17:26:57.264485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.264468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.267861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.267840    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 17:26:57.267985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.267879    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 17:26:57.268979 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.268963    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rw5z\""
Sep 09 17:26:57.278229 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.278204    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"]
Sep 09 17:26:57.308501 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.308468    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.308673 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.308519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.308673 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.308577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgnbq\" (UniqueName: \"kubernetes.io/projected/a01d09bc-37a8-4f02-96b1-d53efbe44e80-kube-api-access-zgnbq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.409809 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.409754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.410020 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.409815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgnbq\" (UniqueName: \"kubernetes.io/projected/a01d09bc-37a8-4f02-96b1-d53efbe44e80-kube-api-access-zgnbq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.410020 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.409897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.410281 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.410245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.410320 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.410267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.417484 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.417448    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"]
Sep 09 17:26:57.419801 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.419778    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.422282 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.422259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgnbq\" (UniqueName: \"kubernetes.io/projected/a01d09bc-37a8-4f02-96b1-d53efbe44e80-kube-api-access-zgnbq\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.422601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.422582    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 17:26:57.422889 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.422873    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 17:26:57.423010 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.422991    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tsgcn\""
Sep 09 17:26:57.423089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.423066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 17:26:57.423427 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.423409    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 17:26:57.440531 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.440506    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"]
Sep 09 17:26:57.510289 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.510249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.510464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.510310    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-manifests\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.510464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.510331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-tmp\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.510464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.510355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlj5\" (UniqueName: \"kubernetes.io/projected/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-kube-api-access-shlj5\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.510464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.510373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-webhook-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.573325 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.573226    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:26:57.611720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.611687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlj5\" (UniqueName: \"kubernetes.io/projected/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-kube-api-access-shlj5\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.611876 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.611728    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-webhook-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.611876 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.611834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.612029 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.611918    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-manifests\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.612029 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.611947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-tmp\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.612399 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.612377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-manifests\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.614333 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.614301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.614612 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.614593    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-tmp\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.614844 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.614824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-webhook-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.621508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.621450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlj5\" (UniqueName: \"kubernetes.io/projected/9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307-kube-api-access-shlj5\") pod \"opendatahub-operator-controller-manager-cdfc9465f-7dhs9\" (UID: \"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.735585 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.735546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:26:57.861890 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.861864    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"]
Sep 09 17:26:57.863853 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:26:57.863827    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01d09bc_37a8_4f02_96b1_d53efbe44e80.slice/crio-77254325c6e938dc7041840e4d80d77481dc03af23055ce86927adddb3e87992 WatchSource:0}: Error finding container 77254325c6e938dc7041840e4d80d77481dc03af23055ce86927adddb3e87992: Status 404 returned error can't find the container with id 77254325c6e938dc7041840e4d80d77481dc03af23055ce86927adddb3e87992
Sep 09 17:26:57.942825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.942787    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2" event={"ID":"a01d09bc-37a8-4f02-96b1-d53efbe44e80","Type":"ContainerStarted","Data":"4773dd123ebe9eaf9ca7ec862a118a165b538e871cf3c4271bf21d8da0d235e8"}
Sep 09 17:26:57.943013 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:57.942854    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2" event={"ID":"a01d09bc-37a8-4f02-96b1-d53efbe44e80","Type":"ContainerStarted","Data":"77254325c6e938dc7041840e4d80d77481dc03af23055ce86927adddb3e87992"}
Sep 09 17:26:58.034085 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:58.034046    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"]
Sep 09 17:26:58.034253 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:26:58.034224    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1ce6fb_a8a9_4eb3_b9fc_c5a7d51f2307.slice/crio-9fdc250784d3c59da6b7dab6ee0d40ea9f57db43dc3236a816d7528014ba9996 WatchSource:0}: Error finding container 9fdc250784d3c59da6b7dab6ee0d40ea9f57db43dc3236a816d7528014ba9996: Status 404 returned error can't find the container with id 9fdc250784d3c59da6b7dab6ee0d40ea9f57db43dc3236a816d7528014ba9996
Sep 09 17:26:58.950206 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:58.950119    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9" event={"ID":"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307","Type":"ContainerStarted","Data":"9fdc250784d3c59da6b7dab6ee0d40ea9f57db43dc3236a816d7528014ba9996"}
Sep 09 17:26:58.951993 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:58.951946    2558 generic.go:356] "Generic (PLEG): container finished" podID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerID="4773dd123ebe9eaf9ca7ec862a118a165b538e871cf3c4271bf21d8da0d235e8" exitCode=0
Sep 09 17:26:58.952115 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:26:58.952038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2" event={"ID":"a01d09bc-37a8-4f02-96b1-d53efbe44e80","Type":"ContainerDied","Data":"4773dd123ebe9eaf9ca7ec862a118a165b538e871cf3c4271bf21d8da0d235e8"}
Sep 09 17:27:01.967338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:01.967305    2558 generic.go:356] "Generic (PLEG): container finished" podID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerID="47e923756379b4698e3e63cdb2e823658b6f0ac44c067115428005731c1f5447" exitCode=0
Sep 09 17:27:01.967772 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:01.967399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2" event={"ID":"a01d09bc-37a8-4f02-96b1-d53efbe44e80","Type":"ContainerDied","Data":"47e923756379b4698e3e63cdb2e823658b6f0ac44c067115428005731c1f5447"}
Sep 09 17:27:01.969099 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:01.969072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9" event={"ID":"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307","Type":"ContainerStarted","Data":"f68327f02a6337f34f8706bf06aba66006df0c0179bbe9f7cff7ee8d96bfe947"}
Sep 09 17:27:02.975238 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:02.975178    2558 generic.go:356] "Generic (PLEG): container finished" podID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerID="920f559df2c8e58d530ca7c32dcee1d36651bf21dd14a38b7218e415cddcfc10" exitCode=0
Sep 09 17:27:02.975633 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:02.975265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2" event={"ID":"a01d09bc-37a8-4f02-96b1-d53efbe44e80","Type":"ContainerDied","Data":"920f559df2c8e58d530ca7c32dcee1d36651bf21dd14a38b7218e415cddcfc10"}
Sep 09 17:27:02.976819 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:02.976797    2558 generic.go:356] "Generic (PLEG): container finished" podID="9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307" containerID="f68327f02a6337f34f8706bf06aba66006df0c0179bbe9f7cff7ee8d96bfe947" exitCode=0
Sep 09 17:27:02.976935 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:02.976843    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9" event={"ID":"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307","Type":"ContainerDied","Data":"f68327f02a6337f34f8706bf06aba66006df0c0179bbe9f7cff7ee8d96bfe947"}
Sep 09 17:27:03.982509 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:03.982466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9" event={"ID":"9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307","Type":"ContainerStarted","Data":"2c97223abc394ab9674069f5b7382269c41a43526ea9cf85a3fee9c5bd45b358"}
Sep 09 17:27:03.982908 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:03.982547    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:27:04.007782 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.007726    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9" podStartSLOduration=2.534956867 podStartE2EDuration="7.007704417s" totalImagesPullingTime="3.428842796s" totalInitContainerRuntime="1.043904754s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:04.005041009 +0000 UTC m=+599.633193757" watchObservedRunningTime="2026-09-09 17:27:04.007704417 +0000 UTC m=+599.635857121"
Sep 09 17:27:04.104570 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.104541    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:27:04.165642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.165606    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-util\") pod \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") "
Sep 09 17:27:04.165801 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.165673    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgnbq\" (UniqueName: \"kubernetes.io/projected/a01d09bc-37a8-4f02-96b1-d53efbe44e80-kube-api-access-zgnbq\") pod \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") "
Sep 09 17:27:04.165801 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.165726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-bundle\") pod \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\" (UID: \"a01d09bc-37a8-4f02-96b1-d53efbe44e80\") "
Sep 09 17:27:04.166446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.166417    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-bundle" (OuterVolumeSpecName: "bundle") pod "a01d09bc-37a8-4f02-96b1-d53efbe44e80" (UID: "a01d09bc-37a8-4f02-96b1-d53efbe44e80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:27:04.167830 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.167808    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01d09bc-37a8-4f02-96b1-d53efbe44e80-kube-api-access-zgnbq" (OuterVolumeSpecName: "kube-api-access-zgnbq") pod "a01d09bc-37a8-4f02-96b1-d53efbe44e80" (UID: "a01d09bc-37a8-4f02-96b1-d53efbe44e80"). InnerVolumeSpecName "kube-api-access-zgnbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:27:04.169892 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.169861    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-util" (OuterVolumeSpecName: "util") pod "a01d09bc-37a8-4f02-96b1-d53efbe44e80" (UID: "a01d09bc-37a8-4f02-96b1-d53efbe44e80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:27:04.266386 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.266302    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:27:04.266386 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.266334    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgnbq\" (UniqueName: \"kubernetes.io/projected/a01d09bc-37a8-4f02-96b1-d53efbe44e80-kube-api-access-zgnbq\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:27:04.266386 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.266344    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01d09bc-37a8-4f02-96b1-d53efbe44e80-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:27:04.724873 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.724840    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:27:04.725186 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.725168    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:27:04.988053 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.987945    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2"
Sep 09 17:27:04.988053 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.987984    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0k7hk2" event={"ID":"a01d09bc-37a8-4f02-96b1-d53efbe44e80","Type":"ContainerDied","Data":"77254325c6e938dc7041840e4d80d77481dc03af23055ce86927adddb3e87992"}
Sep 09 17:27:04.988053 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:04.988029    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77254325c6e938dc7041840e4d80d77481dc03af23055ce86927adddb3e87992"
Sep 09 17:27:08.002746 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:08.002705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c" event={"ID":"b01bb6bd-0677-4010-9acd-3aa55a7d75dc","Type":"ContainerStarted","Data":"3f024db9be8bc86b12e9abdfd20327b0354a0e8f9796ae3dd240ac7113d82225"}
Sep 09 17:27:08.003248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:08.002787    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:27:08.020111 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:08.020064    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c" podStartSLOduration=1.5411775680000002 podStartE2EDuration="15.020045955s" totalImagesPullingTime="13.478868387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:08.019329568 +0000 UTC m=+603.647482270" watchObservedRunningTime="2026-09-09 17:27:08.020045955 +0000 UTC m=+603.648198658"
Sep 09 17:27:14.990523 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:14.990493    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-7dhs9"
Sep 09 17:27:17.245328 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245290    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"]
Sep 09 17:27:17.245776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245537    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerName="extract"
Sep 09 17:27:17.245776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245548    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerName="extract"
Sep 09 17:27:17.245776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245556    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerName="util"
Sep 09 17:27:17.245776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245560    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerName="util"
Sep 09 17:27:17.245776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245571    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerName="pull"
Sep 09 17:27:17.245776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245576    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerName="pull"
Sep 09 17:27:17.245776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.245627    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01d09bc-37a8-4f02-96b1-d53efbe44e80" containerName="extract"
Sep 09 17:27:17.247890 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.247868    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.250611 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.250587    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 17:27:17.250716 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.250630    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 17:27:17.251850 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.251830    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wcppt\""
Sep 09 17:27:17.251944 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.251891    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 17:27:17.251944 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.251902    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 17:27:17.260821 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.260797    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"]
Sep 09 17:27:17.372641 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.372608    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tmp\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.372822 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.372660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8f92\" (UniqueName: \"kubernetes.io/projected/3dff361f-6c81-4d16-9ff2-e81651a5d04a-kube-api-access-k8f92\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.372822 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.372771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tls-certs\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.474052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.474004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tls-certs\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.474225 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.474072    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tmp\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.474225 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.474136    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8f92\" (UniqueName: \"kubernetes.io/projected/3dff361f-6c81-4d16-9ff2-e81651a5d04a-kube-api-access-k8f92\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.474225 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:27:17.474167    2558 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 09 17:27:17.474350 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:27:17.474258    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tls-certs podName:3dff361f-6c81-4d16-9ff2-e81651a5d04a nodeName:}" failed. No retries permitted until 2026-09-09 17:27:17.974235775 +0000 UTC m=+613.602388458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tls-certs") pod "kube-auth-proxy-5d7468889-t9l6t" (UID: "3dff361f-6c81-4d16-9ff2-e81651a5d04a") : secret "kube-auth-proxy-tls" not found
Sep 09 17:27:17.476338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.476315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tmp\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.483120 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.483095    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8f92\" (UniqueName: \"kubernetes.io/projected/3dff361f-6c81-4d16-9ff2-e81651a5d04a-kube-api-access-k8f92\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.978720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.978684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tls-certs\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:17.981082 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:17.981064    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dff361f-6c81-4d16-9ff2-e81651a5d04a-tls-certs\") pod \"kube-auth-proxy-5d7468889-t9l6t\" (UID: \"3dff361f-6c81-4d16-9ff2-e81651a5d04a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:18.156429 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:18.156393    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"
Sep 09 17:27:18.432526 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:18.432501    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d7468889-t9l6t"]
Sep 09 17:27:18.435652 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:27:18.435622    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dff361f_6c81_4d16_9ff2_e81651a5d04a.slice/crio-c7fbbf84004e12e682c63d0d09ec7d10baddfdba8b16933998c2c46ab50ef60a WatchSource:0}: Error finding container c7fbbf84004e12e682c63d0d09ec7d10baddfdba8b16933998c2c46ab50ef60a: Status 404 returned error can't find the container with id c7fbbf84004e12e682c63d0d09ec7d10baddfdba8b16933998c2c46ab50ef60a
Sep 09 17:27:19.009256 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:19.009221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67d8946886-rmz2c"
Sep 09 17:27:19.047822 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:19.047787    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t" event={"ID":"3dff361f-6c81-4d16-9ff2-e81651a5d04a","Type":"ContainerStarted","Data":"c7fbbf84004e12e682c63d0d09ec7d10baddfdba8b16933998c2c46ab50ef60a"}
Sep 09 17:27:22.066374 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:22.066284    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t" event={"ID":"3dff361f-6c81-4d16-9ff2-e81651a5d04a","Type":"ContainerStarted","Data":"ecd5a75636bf172d4827e1edbe0842ebfb3d579b33f33066eecda23891dda10b"}
Sep 09 17:27:22.082933 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:27:22.082876    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d7468889-t9l6t" podStartSLOduration=1.743702383 podStartE2EDuration="5.082860203s" totalImagesPullingTime="3.33915782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:22.082137462 +0000 UTC m=+617.710290165" watchObservedRunningTime="2026-09-09 17:27:22.082860203 +0000 UTC m=+617.711012932"
Sep 09 17:28:32.597919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.597884    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"]
Sep 09 17:28:32.601088 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.601071    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.603634 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.603607    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 17:28:32.603768 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.603613    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rw5z\""
Sep 09 17:28:32.604804 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.604786    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 17:28:32.609788 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.609762    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"]
Sep 09 17:28:32.698170 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.698133    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"]
Sep 09 17:28:32.701331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.701313    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.708980 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.708921    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"]
Sep 09 17:28:32.743170 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.743133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.743343 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.743188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwp5p\" (UniqueName: \"kubernetes.io/projected/eb3f9123-77e3-49eb-934c-04a80d75ef79-kube-api-access-bwp5p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.743343 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.743208    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.800345 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.800308    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"]
Sep 09 17:28:32.805264 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.805235    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:32.816662 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.816638    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"]
Sep 09 17:28:32.844603 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.844568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.844768 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.844616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.844768 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.844668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwp5p\" (UniqueName: \"kubernetes.io/projected/eb3f9123-77e3-49eb-934c-04a80d75ef79-kube-api-access-bwp5p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.844768 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.844695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.844945 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.844770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfx9\" (UniqueName: \"kubernetes.io/projected/be46a100-42b4-4b1f-a55e-56885b2069f7-kube-api-access-wxfx9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.844945 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.844809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.845084 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.844984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.845084 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.845042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.852551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.852496    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwp5p\" (UniqueName: \"kubernetes.io/projected/eb3f9123-77e3-49eb-934c-04a80d75ef79-kube-api-access-bwp5p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.895753 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.895717    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"]
Sep 09 17:28:32.899012 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.898997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:32.907533 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.907494    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"]
Sep 09 17:28:32.910489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.910464    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:32.945640 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.945603    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.945841 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.945654    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:32.945841 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.945689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:32.945841 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.945726    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-kube-api-access-jrv5x\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:32.945841 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.945797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfx9\" (UniqueName: \"kubernetes.io/projected/be46a100-42b4-4b1f-a55e-56885b2069f7-kube-api-access-wxfx9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.946022 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.945848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.946058 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.945998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.946147 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.946133    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:32.954205 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:32.954172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfx9\" (UniqueName: \"kubernetes.io/projected/be46a100-42b4-4b1f-a55e-56885b2069f7-kube-api-access-wxfx9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:33.011070 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.011032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:33.047170 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.047123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4r8b\" (UniqueName: \"kubernetes.io/projected/2ccc5663-f769-476d-a232-467129e2dad9-kube-api-access-c4r8b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.047368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.047183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:33.047368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.047226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:33.047368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.047272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-kube-api-access-jrv5x\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:33.047368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.047302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.047368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.047335    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.048133 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.047822    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:33.048133 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.048097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:33.056328 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.056296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-kube-api-access-jrv5x\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:33.115433 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.115329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:33.148668 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.148632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.148787 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.148673    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.148787 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.148741    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4r8b\" (UniqueName: \"kubernetes.io/projected/2ccc5663-f769-476d-a232-467129e2dad9-kube-api-access-c4r8b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.149096 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.149076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.149149 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.149125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.156833 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.156804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4r8b\" (UniqueName: \"kubernetes.io/projected/2ccc5663-f769-476d-a232-467129e2dad9-kube-api-access-c4r8b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.185025 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.184989    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"]
Sep 09 17:28:33.187255 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:28:33.187225    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3f9123_77e3_49eb_934c_04a80d75ef79.slice/crio-4229f522b170fbe4c178ba8f006fbae2a0b6ac18d8f34d7c62a3a7d6690d5d0b WatchSource:0}: Error finding container 4229f522b170fbe4c178ba8f006fbae2a0b6ac18d8f34d7c62a3a7d6690d5d0b: Status 404 returned error can't find the container with id 4229f522b170fbe4c178ba8f006fbae2a0b6ac18d8f34d7c62a3a7d6690d5d0b
Sep 09 17:28:33.209381 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.209359    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:33.284266 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.284202    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"]
Sep 09 17:28:33.288354 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:28:33.288326    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe46a100_42b4_4b1f_a55e_56885b2069f7.slice/crio-f59eceac49475842c097f4ba8d935babc5fc9e8aa81383e61c0b53a8630fadd4 WatchSource:0}: Error finding container f59eceac49475842c097f4ba8d935babc5fc9e8aa81383e61c0b53a8630fadd4: Status 404 returned error can't find the container with id f59eceac49475842c097f4ba8d935babc5fc9e8aa81383e61c0b53a8630fadd4
Sep 09 17:28:33.368486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.368412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg" event={"ID":"be46a100-42b4-4b1f-a55e-56885b2069f7","Type":"ContainerStarted","Data":"f59eceac49475842c097f4ba8d935babc5fc9e8aa81383e61c0b53a8630fadd4"}
Sep 09 17:28:33.370195 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.370166    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerID="1bcff6dc38b9e6999320f117ea34b68542903b307ea8b43109c79fb18dc88a54" exitCode=0
Sep 09 17:28:33.370284 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.370249    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws" event={"ID":"eb3f9123-77e3-49eb-934c-04a80d75ef79","Type":"ContainerDied","Data":"1bcff6dc38b9e6999320f117ea34b68542903b307ea8b43109c79fb18dc88a54"}
Sep 09 17:28:33.370332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.370293    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws" event={"ID":"eb3f9123-77e3-49eb-934c-04a80d75ef79","Type":"ContainerStarted","Data":"4229f522b170fbe4c178ba8f006fbae2a0b6ac18d8f34d7c62a3a7d6690d5d0b"}
Sep 09 17:28:33.385237 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.385213    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"]
Sep 09 17:28:33.388388 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:28:33.388354    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d6f0fc_07f5_4edc_89c0_1ba543babefa.slice/crio-5e802c48b39cf827c9d82fb650fa58f3ccf58aae81288377e678aa44154e7483 WatchSource:0}: Error finding container 5e802c48b39cf827c9d82fb650fa58f3ccf58aae81288377e678aa44154e7483: Status 404 returned error can't find the container with id 5e802c48b39cf827c9d82fb650fa58f3ccf58aae81288377e678aa44154e7483
Sep 09 17:28:33.481190 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:33.481163    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"]
Sep 09 17:28:33.490864 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:28:33.490835    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccc5663_f769_476d_a232_467129e2dad9.slice/crio-f0fc2cdb3e02c1e83e9bf87365dcfc7723f717b5332f2e1a445a62e9bd0b08c4 WatchSource:0}: Error finding container f0fc2cdb3e02c1e83e9bf87365dcfc7723f717b5332f2e1a445a62e9bd0b08c4: Status 404 returned error can't find the container with id f0fc2cdb3e02c1e83e9bf87365dcfc7723f717b5332f2e1a445a62e9bd0b08c4
Sep 09 17:28:34.375862 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.375831    2558 generic.go:356] "Generic (PLEG): container finished" podID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerID="68b8ba41020f60e7918f29aca1543e66f213ae91695739acf3cc633c2c6a6a68" exitCode=0
Sep 09 17:28:34.376319 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.375918    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg" event={"ID":"be46a100-42b4-4b1f-a55e-56885b2069f7","Type":"ContainerDied","Data":"68b8ba41020f60e7918f29aca1543e66f213ae91695739acf3cc633c2c6a6a68"}
Sep 09 17:28:34.377825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.377798    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerID="a02f7eb636368084aa9196da919f7528a0e737fb326c51e0257790845e40e97f" exitCode=0
Sep 09 17:28:34.377929 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.377879    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws" event={"ID":"eb3f9123-77e3-49eb-934c-04a80d75ef79","Type":"ContainerDied","Data":"a02f7eb636368084aa9196da919f7528a0e737fb326c51e0257790845e40e97f"}
Sep 09 17:28:34.379570 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.379549    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ccc5663-f769-476d-a232-467129e2dad9" containerID="a0926dfcb441b285fcbf243d6ecb68985af12f8488ffcf4d6d4d27da6bfa7bcf" exitCode=0
Sep 09 17:28:34.379671 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.379616    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs" event={"ID":"2ccc5663-f769-476d-a232-467129e2dad9","Type":"ContainerDied","Data":"a0926dfcb441b285fcbf243d6ecb68985af12f8488ffcf4d6d4d27da6bfa7bcf"}
Sep 09 17:28:34.379733 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.379706    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs" event={"ID":"2ccc5663-f769-476d-a232-467129e2dad9","Type":"ContainerStarted","Data":"f0fc2cdb3e02c1e83e9bf87365dcfc7723f717b5332f2e1a445a62e9bd0b08c4"}
Sep 09 17:28:34.381306 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.381281    2558 generic.go:356] "Generic (PLEG): container finished" podID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerID="aa61dd617d8bc1503645c35f500c544ae5d5da11acdfaaedfc05e717994358bb" exitCode=0
Sep 09 17:28:34.381399 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.381314    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj" event={"ID":"c9d6f0fc-07f5-4edc-89c0-1ba543babefa","Type":"ContainerDied","Data":"aa61dd617d8bc1503645c35f500c544ae5d5da11acdfaaedfc05e717994358bb"}
Sep 09 17:28:34.381399 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:34.381345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj" event={"ID":"c9d6f0fc-07f5-4edc-89c0-1ba543babefa","Type":"ContainerStarted","Data":"5e802c48b39cf827c9d82fb650fa58f3ccf58aae81288377e678aa44154e7483"}
Sep 09 17:28:35.389012 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:35.388983    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerID="0ff8766ac828b4dfef75f6f5ca3e9bd1ed54d2239f903c2ff924ba640c520443" exitCode=0
Sep 09 17:28:35.389388 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:35.389118    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws" event={"ID":"eb3f9123-77e3-49eb-934c-04a80d75ef79","Type":"ContainerDied","Data":"0ff8766ac828b4dfef75f6f5ca3e9bd1ed54d2239f903c2ff924ba640c520443"}
Sep 09 17:28:36.399817 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.399777    2558 generic.go:356] "Generic (PLEG): container finished" podID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerID="ec812a7d2f30a5f03f02b2de0d821f1eca3512fcd8e5a7390a1fa9d105a2a40d" exitCode=0
Sep 09 17:28:36.400269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.399864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj" event={"ID":"c9d6f0fc-07f5-4edc-89c0-1ba543babefa","Type":"ContainerDied","Data":"ec812a7d2f30a5f03f02b2de0d821f1eca3512fcd8e5a7390a1fa9d105a2a40d"}
Sep 09 17:28:36.401785 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.401762    2558 generic.go:356] "Generic (PLEG): container finished" podID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerID="8671b4d6ffdbaa34bc6a481eac0f4b110728bb794b2e381c1409ef584cd8aa49" exitCode=0
Sep 09 17:28:36.401935 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.401916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg" event={"ID":"be46a100-42b4-4b1f-a55e-56885b2069f7","Type":"ContainerDied","Data":"8671b4d6ffdbaa34bc6a481eac0f4b110728bb794b2e381c1409ef584cd8aa49"}
Sep 09 17:28:36.403935 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.403910    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ccc5663-f769-476d-a232-467129e2dad9" containerID="7c66b00fe83eb6436e7addfe9f3239d8adbaece8e18842de7025bcf94cc8037f" exitCode=0
Sep 09 17:28:36.404053 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.403984    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs" event={"ID":"2ccc5663-f769-476d-a232-467129e2dad9","Type":"ContainerDied","Data":"7c66b00fe83eb6436e7addfe9f3239d8adbaece8e18842de7025bcf94cc8037f"}
Sep 09 17:28:36.547346 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.547313    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:36.676348 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.676317    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-util\") pod \"eb3f9123-77e3-49eb-934c-04a80d75ef79\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") "
Sep 09 17:28:36.676476 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.676404    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-bundle\") pod \"eb3f9123-77e3-49eb-934c-04a80d75ef79\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") "
Sep 09 17:28:36.676476 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.676424    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwp5p\" (UniqueName: \"kubernetes.io/projected/eb3f9123-77e3-49eb-934c-04a80d75ef79-kube-api-access-bwp5p\") pod \"eb3f9123-77e3-49eb-934c-04a80d75ef79\" (UID: \"eb3f9123-77e3-49eb-934c-04a80d75ef79\") "
Sep 09 17:28:36.676888 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.676854    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-bundle" (OuterVolumeSpecName: "bundle") pod "eb3f9123-77e3-49eb-934c-04a80d75ef79" (UID: "eb3f9123-77e3-49eb-934c-04a80d75ef79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:36.678525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.678500    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3f9123-77e3-49eb-934c-04a80d75ef79-kube-api-access-bwp5p" (OuterVolumeSpecName: "kube-api-access-bwp5p") pod "eb3f9123-77e3-49eb-934c-04a80d75ef79" (UID: "eb3f9123-77e3-49eb-934c-04a80d75ef79"). InnerVolumeSpecName "kube-api-access-bwp5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:28:36.681685 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.681638    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-util" (OuterVolumeSpecName: "util") pod "eb3f9123-77e3-49eb-934c-04a80d75ef79" (UID: "eb3f9123-77e3-49eb-934c-04a80d75ef79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:36.777586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.777546    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:36.777586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.777578    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb3f9123-77e3-49eb-934c-04a80d75ef79-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:36.777586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:36.777588    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwp5p\" (UniqueName: \"kubernetes.io/projected/eb3f9123-77e3-49eb-934c-04a80d75ef79-kube-api-access-bwp5p\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:37.410770 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.410732    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ccc5663-f769-476d-a232-467129e2dad9" containerID="2f5ffc46fbc89a694305f9d4ad3d09702da5d78f80b7077ae7bb27826fbdd2b0" exitCode=0
Sep 09 17:28:37.411169 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.410820    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs" event={"ID":"2ccc5663-f769-476d-a232-467129e2dad9","Type":"ContainerDied","Data":"2f5ffc46fbc89a694305f9d4ad3d09702da5d78f80b7077ae7bb27826fbdd2b0"}
Sep 09 17:28:37.412940 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.412914    2558 generic.go:356] "Generic (PLEG): container finished" podID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerID="d685d445bc9928f4611605e869bb2de65ff7ac3faa4ddd613431963c1f4f437b" exitCode=0
Sep 09 17:28:37.413072 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.413014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj" event={"ID":"c9d6f0fc-07f5-4edc-89c0-1ba543babefa","Type":"ContainerDied","Data":"d685d445bc9928f4611605e869bb2de65ff7ac3faa4ddd613431963c1f4f437b"}
Sep 09 17:28:37.415160 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.415141    2558 generic.go:356] "Generic (PLEG): container finished" podID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerID="dc679841ead641dd13741ba9fc196bad5d9992201a2b4226db5c260343013a5c" exitCode=0
Sep 09 17:28:37.415240 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.415220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg" event={"ID":"be46a100-42b4-4b1f-a55e-56885b2069f7","Type":"ContainerDied","Data":"dc679841ead641dd13741ba9fc196bad5d9992201a2b4226db5c260343013a5c"}
Sep 09 17:28:37.416867 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.416851    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws"
Sep 09 17:28:37.416974 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.416851    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l75ws" event={"ID":"eb3f9123-77e3-49eb-934c-04a80d75ef79","Type":"ContainerDied","Data":"4229f522b170fbe4c178ba8f006fbae2a0b6ac18d8f34d7c62a3a7d6690d5d0b"}
Sep 09 17:28:37.417042 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:37.416986    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4229f522b170fbe4c178ba8f006fbae2a0b6ac18d8f34d7c62a3a7d6690d5d0b"
Sep 09 17:28:38.557882 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.557854    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:38.593385 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.593360    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:28:38.597486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.597463    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:38.693993 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.693933    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxfx9\" (UniqueName: \"kubernetes.io/projected/be46a100-42b4-4b1f-a55e-56885b2069f7-kube-api-access-wxfx9\") pod \"be46a100-42b4-4b1f-a55e-56885b2069f7\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") "
Sep 09 17:28:38.694167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694004    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-util\") pod \"2ccc5663-f769-476d-a232-467129e2dad9\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") "
Sep 09 17:28:38.694167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694054    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-util\") pod \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") "
Sep 09 17:28:38.694167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-kube-api-access-jrv5x\") pod \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") "
Sep 09 17:28:38.694167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694114    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-bundle\") pod \"be46a100-42b4-4b1f-a55e-56885b2069f7\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") "
Sep 09 17:28:38.694167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694135    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-bundle\") pod \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\" (UID: \"c9d6f0fc-07f5-4edc-89c0-1ba543babefa\") "
Sep 09 17:28:38.694167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694155    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4r8b\" (UniqueName: \"kubernetes.io/projected/2ccc5663-f769-476d-a232-467129e2dad9-kube-api-access-c4r8b\") pod \"2ccc5663-f769-476d-a232-467129e2dad9\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") "
Sep 09 17:28:38.694453 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694190    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-util\") pod \"be46a100-42b4-4b1f-a55e-56885b2069f7\" (UID: \"be46a100-42b4-4b1f-a55e-56885b2069f7\") "
Sep 09 17:28:38.694453 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694225    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-bundle\") pod \"2ccc5663-f769-476d-a232-467129e2dad9\" (UID: \"2ccc5663-f769-476d-a232-467129e2dad9\") "
Sep 09 17:28:38.695248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.694877    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-bundle" (OuterVolumeSpecName: "bundle") pod "be46a100-42b4-4b1f-a55e-56885b2069f7" (UID: "be46a100-42b4-4b1f-a55e-56885b2069f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:38.695248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.695036    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-bundle" (OuterVolumeSpecName: "bundle") pod "2ccc5663-f769-476d-a232-467129e2dad9" (UID: "2ccc5663-f769-476d-a232-467129e2dad9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:38.695248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.695198    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-bundle" (OuterVolumeSpecName: "bundle") pod "c9d6f0fc-07f5-4edc-89c0-1ba543babefa" (UID: "c9d6f0fc-07f5-4edc-89c0-1ba543babefa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:38.696639 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.696607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccc5663-f769-476d-a232-467129e2dad9-kube-api-access-c4r8b" (OuterVolumeSpecName: "kube-api-access-c4r8b") pod "2ccc5663-f769-476d-a232-467129e2dad9" (UID: "2ccc5663-f769-476d-a232-467129e2dad9"). InnerVolumeSpecName "kube-api-access-c4r8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:28:38.696834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.696810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be46a100-42b4-4b1f-a55e-56885b2069f7-kube-api-access-wxfx9" (OuterVolumeSpecName: "kube-api-access-wxfx9") pod "be46a100-42b4-4b1f-a55e-56885b2069f7" (UID: "be46a100-42b4-4b1f-a55e-56885b2069f7"). InnerVolumeSpecName "kube-api-access-wxfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:28:38.696895 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.696872    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-kube-api-access-jrv5x" (OuterVolumeSpecName: "kube-api-access-jrv5x") pod "c9d6f0fc-07f5-4edc-89c0-1ba543babefa" (UID: "c9d6f0fc-07f5-4edc-89c0-1ba543babefa"). InnerVolumeSpecName "kube-api-access-jrv5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:28:38.700395 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.700371    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-util" (OuterVolumeSpecName: "util") pod "be46a100-42b4-4b1f-a55e-56885b2069f7" (UID: "be46a100-42b4-4b1f-a55e-56885b2069f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:38.703497 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.703469    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-util" (OuterVolumeSpecName: "util") pod "2ccc5663-f769-476d-a232-467129e2dad9" (UID: "2ccc5663-f769-476d-a232-467129e2dad9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:38.703680 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.703664    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-util" (OuterVolumeSpecName: "util") pod "c9d6f0fc-07f5-4edc-89c0-1ba543babefa" (UID: "c9d6f0fc-07f5-4edc-89c0-1ba543babefa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:28:38.795042 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795006    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795042 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795037    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-kube-api-access-jrv5x\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795042 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795047    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795059    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9d6f0fc-07f5-4edc-89c0-1ba543babefa-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795075    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4r8b\" (UniqueName: \"kubernetes.io/projected/2ccc5663-f769-476d-a232-467129e2dad9-kube-api-access-c4r8b\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795083    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be46a100-42b4-4b1f-a55e-56885b2069f7-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795091    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-bundle\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795099    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxfx9\" (UniqueName: \"kubernetes.io/projected/be46a100-42b4-4b1f-a55e-56885b2069f7-kube-api-access-wxfx9\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:38.795274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:38.795106    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccc5663-f769-476d-a232-467129e2dad9-util\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:28:39.429535 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.429506    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg"
Sep 09 17:28:39.429731 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.429538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czb2vg" event={"ID":"be46a100-42b4-4b1f-a55e-56885b2069f7","Type":"ContainerDied","Data":"f59eceac49475842c097f4ba8d935babc5fc9e8aa81383e61c0b53a8630fadd4"}
Sep 09 17:28:39.429731 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.429587    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f59eceac49475842c097f4ba8d935babc5fc9e8aa81383e61c0b53a8630fadd4"
Sep 09 17:28:39.431724 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.431699    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs" event={"ID":"2ccc5663-f769-476d-a232-467129e2dad9","Type":"ContainerDied","Data":"f0fc2cdb3e02c1e83e9bf87365dcfc7723f717b5332f2e1a445a62e9bd0b08c4"}
Sep 09 17:28:39.431834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.431739    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kj6qs"
Sep 09 17:28:39.431834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.431741    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0fc2cdb3e02c1e83e9bf87365dcfc7723f717b5332f2e1a445a62e9bd0b08c4"
Sep 09 17:28:39.433907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.433881    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj" event={"ID":"c9d6f0fc-07f5-4edc-89c0-1ba543babefa","Type":"ContainerDied","Data":"5e802c48b39cf827c9d82fb650fa58f3ccf58aae81288377e678aa44154e7483"}
Sep 09 17:28:39.434034 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.433918    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e802c48b39cf827c9d82fb650fa58f3ccf58aae81288377e678aa44154e7483"
Sep 09 17:28:39.434075 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:28:39.434032    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awxwfj"
Sep 09 17:29:48.881332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881297    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xr5rm"]
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881545    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881555    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881565    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881571    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881576    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccc5663-f769-476d-a232-467129e2dad9" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881582    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccc5663-f769-476d-a232-467129e2dad9" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881587    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881591    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881600    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccc5663-f769-476d-a232-467129e2dad9" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881604    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccc5663-f769-476d-a232-467129e2dad9" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881611    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881616    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881621    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccc5663-f769-476d-a232-467129e2dad9" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881626    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccc5663-f769-476d-a232-467129e2dad9" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881631    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881636    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881642    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881647    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881654    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881658    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881665    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881671    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerName="util"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881676    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881681    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerName="pull"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881721    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3f9123-77e3-49eb-934c-04a80d75ef79" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881728    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ccc5663-f769-476d-a232-467129e2dad9" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881733    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d6f0fc-07f5-4edc-89c0-1ba543babefa" containerName="extract"
Sep 09 17:29:48.881781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.881739    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="be46a100-42b4-4b1f-a55e-56885b2069f7" containerName="extract"
Sep 09 17:29:48.884550 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.884531    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:48.887276 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.887255    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 17:29:48.887404 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.887266    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q6dcm\""
Sep 09 17:29:48.887404 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.887255    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 17:29:48.888272 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.888234    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 17:29:48.892338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.892318    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xr5rm"]
Sep 09 17:29:48.975976 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:48.975929    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xr5rm"]
Sep 09 17:29:49.020878 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.020846    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-config-file\") pod \"limitador-limitador-8588749977-xr5rm\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") " pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:49.021054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.020907    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxrb\" (UniqueName: \"kubernetes.io/projected/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-kube-api-access-5qxrb\") pod \"limitador-limitador-8588749977-xr5rm\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") " pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:49.121268 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.121229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxrb\" (UniqueName: \"kubernetes.io/projected/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-kube-api-access-5qxrb\") pod \"limitador-limitador-8588749977-xr5rm\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") " pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:49.121268 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.121276    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-config-file\") pod \"limitador-limitador-8588749977-xr5rm\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") " pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:49.121893 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.121874    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-config-file\") pod \"limitador-limitador-8588749977-xr5rm\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") " pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:49.129651 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.129627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxrb\" (UniqueName: \"kubernetes.io/projected/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-kube-api-access-5qxrb\") pod \"limitador-limitador-8588749977-xr5rm\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") " pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:49.194706 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.194671    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:49.466569 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.466543    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xr5rm"]
Sep 09 17:29:49.468363 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:29:49.468334    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6beae4_c5a9_44a9_90a8_00a5e658bf45.slice/crio-b29aecfe3f5827bbf5770f38742969b07833b8b14eac185231a9e4700e7d5855 WatchSource:0}: Error finding container b29aecfe3f5827bbf5770f38742969b07833b8b14eac185231a9e4700e7d5855: Status 404 returned error can't find the container with id b29aecfe3f5827bbf5770f38742969b07833b8b14eac185231a9e4700e7d5855
Sep 09 17:29:49.682936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.682900    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mk5rl"]
Sep 09 17:29:49.687227 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.687204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl"
Sep 09 17:29:49.690114 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.690087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jhjgt\""
Sep 09 17:29:49.692348 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.692322    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mk5rl"]
Sep 09 17:29:49.753080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.752976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm" event={"ID":"dc6beae4-c5a9-44a9-90a8-00a5e658bf45","Type":"ContainerStarted","Data":"b29aecfe3f5827bbf5770f38742969b07833b8b14eac185231a9e4700e7d5855"}
Sep 09 17:29:49.827273 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.827241    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sr5s\" (UniqueName: \"kubernetes.io/projected/c7a4129f-cc5d-4c87-820e-23f18ef20a50-kube-api-access-5sr5s\") pod \"authorino-7bcfd8f9cd-mk5rl\" (UID: \"c7a4129f-cc5d-4c87-820e-23f18ef20a50\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl"
Sep 09 17:29:49.852116 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.852089    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-f67s8"]
Sep 09 17:29:49.855119 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.855104    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f67s8"
Sep 09 17:29:49.861832 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.861806    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f67s8"]
Sep 09 17:29:49.928510 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.928475    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httpb\" (UniqueName: \"kubernetes.io/projected/d4f0a12a-ee15-479b-a640-1bd2d3a0a825-kube-api-access-httpb\") pod \"authorino-59c974bd48-f67s8\" (UID: \"d4f0a12a-ee15-479b-a640-1bd2d3a0a825\") " pod="kuadrant-system/authorino-59c974bd48-f67s8"
Sep 09 17:29:49.928510 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.928510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sr5s\" (UniqueName: \"kubernetes.io/projected/c7a4129f-cc5d-4c87-820e-23f18ef20a50-kube-api-access-5sr5s\") pod \"authorino-7bcfd8f9cd-mk5rl\" (UID: \"c7a4129f-cc5d-4c87-820e-23f18ef20a50\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl"
Sep 09 17:29:49.936630 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.936601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sr5s\" (UniqueName: \"kubernetes.io/projected/c7a4129f-cc5d-4c87-820e-23f18ef20a50-kube-api-access-5sr5s\") pod \"authorino-7bcfd8f9cd-mk5rl\" (UID: \"c7a4129f-cc5d-4c87-820e-23f18ef20a50\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl"
Sep 09 17:29:49.997525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:49.997488    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl"
Sep 09 17:29:50.029971 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:50.029926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httpb\" (UniqueName: \"kubernetes.io/projected/d4f0a12a-ee15-479b-a640-1bd2d3a0a825-kube-api-access-httpb\") pod \"authorino-59c974bd48-f67s8\" (UID: \"d4f0a12a-ee15-479b-a640-1bd2d3a0a825\") " pod="kuadrant-system/authorino-59c974bd48-f67s8"
Sep 09 17:29:50.037776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:50.037744    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-httpb\" (UniqueName: \"kubernetes.io/projected/d4f0a12a-ee15-479b-a640-1bd2d3a0a825-kube-api-access-httpb\") pod \"authorino-59c974bd48-f67s8\" (UID: \"d4f0a12a-ee15-479b-a640-1bd2d3a0a825\") " pod="kuadrant-system/authorino-59c974bd48-f67s8"
Sep 09 17:29:50.164757 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:50.164722    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f67s8"
Sep 09 17:29:50.264742 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:50.264713    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mk5rl"]
Sep 09 17:29:50.266515 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:29:50.266474    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7a4129f_cc5d_4c87_820e_23f18ef20a50.slice/crio-5b6caec50f5e7aaa23c7727f2f1e5db6b990a0e99d50637638e471678aace3d9 WatchSource:0}: Error finding container 5b6caec50f5e7aaa23c7727f2f1e5db6b990a0e99d50637638e471678aace3d9: Status 404 returned error can't find the container with id 5b6caec50f5e7aaa23c7727f2f1e5db6b990a0e99d50637638e471678aace3d9
Sep 09 17:29:50.434502 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:50.434479    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f67s8"]
Sep 09 17:29:50.436178 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:29:50.436152    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4f0a12a_ee15_479b_a640_1bd2d3a0a825.slice/crio-d75417eb22285160a9896bf6ab6fbf5e3ac8d438e0931897895451ab7dadd387 WatchSource:0}: Error finding container d75417eb22285160a9896bf6ab6fbf5e3ac8d438e0931897895451ab7dadd387: Status 404 returned error can't find the container with id d75417eb22285160a9896bf6ab6fbf5e3ac8d438e0931897895451ab7dadd387
Sep 09 17:29:50.760676 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:50.760635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f67s8" event={"ID":"d4f0a12a-ee15-479b-a640-1bd2d3a0a825","Type":"ContainerStarted","Data":"d75417eb22285160a9896bf6ab6fbf5e3ac8d438e0931897895451ab7dadd387"}
Sep 09 17:29:50.761948 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:50.761917    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl" event={"ID":"c7a4129f-cc5d-4c87-820e-23f18ef20a50","Type":"ContainerStarted","Data":"5b6caec50f5e7aaa23c7727f2f1e5db6b990a0e99d50637638e471678aace3d9"}
Sep 09 17:29:52.776996 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:52.776942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm" event={"ID":"dc6beae4-c5a9-44a9-90a8-00a5e658bf45","Type":"ContainerStarted","Data":"4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb"}
Sep 09 17:29:52.777455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:52.777229    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:29:52.793791 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:52.793727    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm" podStartSLOduration=2.322103256 podStartE2EDuration="4.793707488s" totalImagesPullingTime="2.471604232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:52.792662424 +0000 UTC m=+768.420815128" watchObservedRunningTime="2026-09-09 17:29:52.793707488 +0000 UTC m=+768.421860190"
Sep 09 17:29:53.783626 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:53.783591    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f67s8" event={"ID":"d4f0a12a-ee15-479b-a640-1bd2d3a0a825","Type":"ContainerStarted","Data":"43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9"}
Sep 09 17:29:53.785240 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:53.785208    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl" event={"ID":"c7a4129f-cc5d-4c87-820e-23f18ef20a50","Type":"ContainerStarted","Data":"5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef"}
Sep 09 17:29:53.798748 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:53.798703    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-f67s8" podStartSLOduration=2.194713885 podStartE2EDuration="4.798690143s" totalImagesPullingTime="2.603976258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:53.797905588 +0000 UTC m=+769.426058291" watchObservedRunningTime="2026-09-09 17:29:53.798690143 +0000 UTC m=+769.426842866"
Sep 09 17:29:53.815114 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:53.815067    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl" podStartSLOduration=2.037114991 podStartE2EDuration="4.815045214s" totalImagesPullingTime="2.777930223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:53.814122056 +0000 UTC m=+769.442274760" watchObservedRunningTime="2026-09-09 17:29:53.815045214 +0000 UTC m=+769.443197916"
Sep 09 17:29:53.837877 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:53.837846    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mk5rl"]
Sep 09 17:29:55.795941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:55.795872    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl" podUID="c7a4129f-cc5d-4c87-820e-23f18ef20a50" containerName="authorino" containerID="cri-o://5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef" gracePeriod=30
Sep 09 17:29:56.038411 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.038385    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl"
Sep 09 17:29:56.180378 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.180345    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sr5s\" (UniqueName: \"kubernetes.io/projected/c7a4129f-cc5d-4c87-820e-23f18ef20a50-kube-api-access-5sr5s\") pod \"c7a4129f-cc5d-4c87-820e-23f18ef20a50\" (UID: \"c7a4129f-cc5d-4c87-820e-23f18ef20a50\") "
Sep 09 17:29:56.182501 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.182471    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a4129f-cc5d-4c87-820e-23f18ef20a50-kube-api-access-5sr5s" (OuterVolumeSpecName: "kube-api-access-5sr5s") pod "c7a4129f-cc5d-4c87-820e-23f18ef20a50" (UID: "c7a4129f-cc5d-4c87-820e-23f18ef20a50"). InnerVolumeSpecName "kube-api-access-5sr5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:29:56.281833 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.281788    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sr5s\" (UniqueName: \"kubernetes.io/projected/c7a4129f-cc5d-4c87-820e-23f18ef20a50-kube-api-access-5sr5s\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:29:56.801212 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.801175    2558 generic.go:356] "Generic (PLEG): container finished" podID="c7a4129f-cc5d-4c87-820e-23f18ef20a50" containerID="5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef" exitCode=0
Sep 09 17:29:56.801682 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.801221    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl"
Sep 09 17:29:56.801682 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.801262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl" event={"ID":"c7a4129f-cc5d-4c87-820e-23f18ef20a50","Type":"ContainerDied","Data":"5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef"}
Sep 09 17:29:56.801682 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.801304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mk5rl" event={"ID":"c7a4129f-cc5d-4c87-820e-23f18ef20a50","Type":"ContainerDied","Data":"5b6caec50f5e7aaa23c7727f2f1e5db6b990a0e99d50637638e471678aace3d9"}
Sep 09 17:29:56.801682 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.801321    2558 scope.go:117] "RemoveContainer" containerID="5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef"
Sep 09 17:29:56.809987 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.809967    2558 scope.go:117] "RemoveContainer" containerID="5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef"
Sep 09 17:29:56.810265 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:29:56.810248    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef\": container with ID starting with 5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef not found: ID does not exist" containerID="5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef"
Sep 09 17:29:56.810314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.810273    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef"} err="failed to get container status \"5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef\": rpc error: code = NotFound desc = could not find container \"5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef\": container with ID starting with 5dab6c45d1637e8c436ea9242cdb25edf2cf10272d24450f2fcc423f5d7986ef not found: ID does not exist"
Sep 09 17:29:56.821676 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.821652    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mk5rl"]
Sep 09 17:29:56.823299 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:56.823278    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mk5rl"]
Sep 09 17:29:58.815404 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:29:58.815372    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a4129f-cc5d-4c87-820e-23f18ef20a50" path="/var/lib/kubelet/pods/c7a4129f-cc5d-4c87-820e-23f18ef20a50/volumes"
Sep 09 17:30:02.504989 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:02.504938    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xr5rm"]
Sep 09 17:30:02.505424 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:02.505209    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm" podUID="dc6beae4-c5a9-44a9-90a8-00a5e658bf45" containerName="limitador" containerID="cri-o://4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb" gracePeriod=30
Sep 09 17:30:02.505977 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:02.505933    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:30:03.050834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.050804    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:30:03.135284 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.135189    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qxrb\" (UniqueName: \"kubernetes.io/projected/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-kube-api-access-5qxrb\") pod \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") "
Sep 09 17:30:03.135284 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.135239    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-config-file\") pod \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\" (UID: \"dc6beae4-c5a9-44a9-90a8-00a5e658bf45\") "
Sep 09 17:30:03.135628 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.135602    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-config-file" (OuterVolumeSpecName: "config-file") pod "dc6beae4-c5a9-44a9-90a8-00a5e658bf45" (UID: "dc6beae4-c5a9-44a9-90a8-00a5e658bf45"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:30:03.137292 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.137266    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-kube-api-access-5qxrb" (OuterVolumeSpecName: "kube-api-access-5qxrb") pod "dc6beae4-c5a9-44a9-90a8-00a5e658bf45" (UID: "dc6beae4-c5a9-44a9-90a8-00a5e658bf45"). InnerVolumeSpecName "kube-api-access-5qxrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:30:03.235881 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.235842    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qxrb\" (UniqueName: \"kubernetes.io/projected/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-kube-api-access-5qxrb\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:03.235881 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.235874    2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dc6beae4-c5a9-44a9-90a8-00a5e658bf45-config-file\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:03.834095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.834063    2558 generic.go:356] "Generic (PLEG): container finished" podID="dc6beae4-c5a9-44a9-90a8-00a5e658bf45" containerID="4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb" exitCode=0
Sep 09 17:30:03.834483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.834108    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm"
Sep 09 17:30:03.834483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.834151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm" event={"ID":"dc6beae4-c5a9-44a9-90a8-00a5e658bf45","Type":"ContainerDied","Data":"4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb"}
Sep 09 17:30:03.834483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.834198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xr5rm" event={"ID":"dc6beae4-c5a9-44a9-90a8-00a5e658bf45","Type":"ContainerDied","Data":"b29aecfe3f5827bbf5770f38742969b07833b8b14eac185231a9e4700e7d5855"}
Sep 09 17:30:03.834483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.834219    2558 scope.go:117] "RemoveContainer" containerID="4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb"
Sep 09 17:30:03.842649 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.842632    2558 scope.go:117] "RemoveContainer" containerID="4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb"
Sep 09 17:30:03.842890 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:30:03.842869    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb\": container with ID starting with 4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb not found: ID does not exist" containerID="4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb"
Sep 09 17:30:03.842981 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.842903    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb"} err="failed to get container status \"4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb\": rpc error: code = NotFound desc = could not find container \"4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb\": container with ID starting with 4eb1aae0c0a0eaa7af245d16fedb4f4d39faf417b8f39b25675d28f1cec3babb not found: ID does not exist"
Sep 09 17:30:03.854243 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.854213    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xr5rm"]
Sep 09 17:30:03.855856 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:03.855834    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xr5rm"]
Sep 09 17:30:04.815217 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:04.815184    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6beae4-c5a9-44a9-90a8-00a5e658bf45" path="/var/lib/kubelet/pods/dc6beae4-c5a9-44a9-90a8-00a5e658bf45/volumes"
Sep 09 17:30:08.212464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.212430    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"]
Sep 09 17:30:08.212836 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.212820    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6beae4-c5a9-44a9-90a8-00a5e658bf45" containerName="limitador"
Sep 09 17:30:08.212879 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.212840    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6beae4-c5a9-44a9-90a8-00a5e658bf45" containerName="limitador"
Sep 09 17:30:08.212879 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.212874    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a4129f-cc5d-4c87-820e-23f18ef20a50" containerName="authorino"
Sep 09 17:30:08.212946 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.212883    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a4129f-cc5d-4c87-820e-23f18ef20a50" containerName="authorino"
Sep 09 17:30:08.213007 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.212965    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6beae4-c5a9-44a9-90a8-00a5e658bf45" containerName="limitador"
Sep 09 17:30:08.213007 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.212976    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a4129f-cc5d-4c87-820e-23f18ef20a50" containerName="authorino"
Sep 09 17:30:08.217347 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.217331    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.220101 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.220077    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-p4fcw\""
Sep 09 17:30:08.220934 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.220910    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 17:30:08.221075 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.221015    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 17:30:08.221075 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.221030    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 17:30:08.222097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.222075    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"]
Sep 09 17:30:08.376998 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.376943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82pg\" (UniqueName: \"kubernetes.io/projected/2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe-kube-api-access-b82pg\") pod \"postgres-69fb4594cf-sl7kq\" (UID: \"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.377152 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.377056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe-data\") pod \"postgres-69fb4594cf-sl7kq\" (UID: \"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.478275 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.478183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe-data\") pod \"postgres-69fb4594cf-sl7kq\" (UID: \"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.478275 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.478243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82pg\" (UniqueName: \"kubernetes.io/projected/2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe-kube-api-access-b82pg\") pod \"postgres-69fb4594cf-sl7kq\" (UID: \"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.478636 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.478614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe-data\") pod \"postgres-69fb4594cf-sl7kq\" (UID: \"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.486655 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.486630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82pg\" (UniqueName: \"kubernetes.io/projected/2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe-kube-api-access-b82pg\") pod \"postgres-69fb4594cf-sl7kq\" (UID: \"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.529453 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.529414    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:08.800321 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.800297    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"]
Sep 09 17:30:08.802173 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:30:08.802138    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca2bd2a_96d9_4cc2_bf87_006af20f2cfe.slice/crio-ec82e605b462f9cb2cdde5d6d5754f06d75bc9668b5fe023561a854eab5b9dce WatchSource:0}: Error finding container ec82e605b462f9cb2cdde5d6d5754f06d75bc9668b5fe023561a854eab5b9dce: Status 404 returned error can't find the container with id ec82e605b462f9cb2cdde5d6d5754f06d75bc9668b5fe023561a854eab5b9dce
Sep 09 17:30:08.856776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:08.856733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq" event={"ID":"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe","Type":"ContainerStarted","Data":"ec82e605b462f9cb2cdde5d6d5754f06d75bc9668b5fe023561a854eab5b9dce"}
Sep 09 17:30:13.888579 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:13.888486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq" event={"ID":"2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe","Type":"ContainerStarted","Data":"99689e511bf5f787a07985c5167771d3c9f83cdb3ab44f82cbc75e70d4184d6c"}
Sep 09 17:30:13.889006 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:13.888587    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:13.904638 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:13.904589    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq" podStartSLOduration=1.126605754 podStartE2EDuration="5.904576409s" totalImagesPullingTime="4.777970655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:13.9034618 +0000 UTC m=+789.531614503" watchObservedRunningTime="2026-09-09 17:30:13.904576409 +0000 UTC m=+789.532729109"
Sep 09 17:30:19.922268 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:19.922226    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sl7kq"
Sep 09 17:30:20.448306 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.448267    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-d42fs"]
Sep 09 17:30:20.454276 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.454259    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-d42fs"
Sep 09 17:30:20.458282 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.458256    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-d42fs"]
Sep 09 17:30:20.467165 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.467143    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crjr\" (UniqueName: \"kubernetes.io/projected/0a30b228-1b96-47d9-98a6-e70dcda57b05-kube-api-access-2crjr\") pod \"authorino-557f876f69-d42fs\" (UID: \"0a30b228-1b96-47d9-98a6-e70dcda57b05\") " pod="kuadrant-system/authorino-557f876f69-d42fs"
Sep 09 17:30:20.568425 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.568381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crjr\" (UniqueName: \"kubernetes.io/projected/0a30b228-1b96-47d9-98a6-e70dcda57b05-kube-api-access-2crjr\") pod \"authorino-557f876f69-d42fs\" (UID: \"0a30b228-1b96-47d9-98a6-e70dcda57b05\") " pod="kuadrant-system/authorino-557f876f69-d42fs"
Sep 09 17:30:20.576758 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.576731    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crjr\" (UniqueName: \"kubernetes.io/projected/0a30b228-1b96-47d9-98a6-e70dcda57b05-kube-api-access-2crjr\") pod \"authorino-557f876f69-d42fs\" (UID: \"0a30b228-1b96-47d9-98a6-e70dcda57b05\") " pod="kuadrant-system/authorino-557f876f69-d42fs"
Sep 09 17:30:20.680248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.680210    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-d42fs"]
Sep 09 17:30:20.680435 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.680424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-d42fs"
Sep 09 17:30:20.705210 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.705147    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-744949959-jfbbs"]
Sep 09 17:30:20.710373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.710350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-744949959-jfbbs"
Sep 09 17:30:20.714941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.714912    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-744949959-jfbbs"]
Sep 09 17:30:20.769484 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.769453    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgt4\" (UniqueName: \"kubernetes.io/projected/a3e37c45-12fc-4e38-991c-8c43ca84bac9-kube-api-access-mcgt4\") pod \"authorino-744949959-jfbbs\" (UID: \"a3e37c45-12fc-4e38-991c-8c43ca84bac9\") " pod="kuadrant-system/authorino-744949959-jfbbs"
Sep 09 17:30:20.870603 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.870563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgt4\" (UniqueName: \"kubernetes.io/projected/a3e37c45-12fc-4e38-991c-8c43ca84bac9-kube-api-access-mcgt4\") pod \"authorino-744949959-jfbbs\" (UID: \"a3e37c45-12fc-4e38-991c-8c43ca84bac9\") " pod="kuadrant-system/authorino-744949959-jfbbs"
Sep 09 17:30:20.880228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.880200    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgt4\" (UniqueName: \"kubernetes.io/projected/a3e37c45-12fc-4e38-991c-8c43ca84bac9-kube-api-access-mcgt4\") pod \"authorino-744949959-jfbbs\" (UID: \"a3e37c45-12fc-4e38-991c-8c43ca84bac9\") " pod="kuadrant-system/authorino-744949959-jfbbs"
Sep 09 17:30:20.953498 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:20.953464    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-d42fs"]
Sep 09 17:30:20.955856 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:30:20.955792    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a30b228_1b96_47d9_98a6_e70dcda57b05.slice/crio-758e7813079421bcf7d4849f47eda4bae671341cbb03f72747cb14e7092f588d WatchSource:0}: Error finding container 758e7813079421bcf7d4849f47eda4bae671341cbb03f72747cb14e7092f588d: Status 404 returned error can't find the container with id 758e7813079421bcf7d4849f47eda4bae671341cbb03f72747cb14e7092f588d
Sep 09 17:30:21.021619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.021575    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-744949959-jfbbs"
Sep 09 17:30:21.290153 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.290062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-744949959-jfbbs"]
Sep 09 17:30:21.292275 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:30:21.292249    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e37c45_12fc_4e38_991c_8c43ca84bac9.slice/crio-0f00ae70deb9b27c32dda0b66c0ccd42eb1d3aa7d764c68fb5211a05a37b8de9 WatchSource:0}: Error finding container 0f00ae70deb9b27c32dda0b66c0ccd42eb1d3aa7d764c68fb5211a05a37b8de9: Status 404 returned error can't find the container with id 0f00ae70deb9b27c32dda0b66c0ccd42eb1d3aa7d764c68fb5211a05a37b8de9
Sep 09 17:30:21.927657 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.927627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-744949959-jfbbs" event={"ID":"a3e37c45-12fc-4e38-991c-8c43ca84bac9","Type":"ContainerStarted","Data":"165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6"}
Sep 09 17:30:21.927842 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.927677    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-744949959-jfbbs" event={"ID":"a3e37c45-12fc-4e38-991c-8c43ca84bac9","Type":"ContainerStarted","Data":"0f00ae70deb9b27c32dda0b66c0ccd42eb1d3aa7d764c68fb5211a05a37b8de9"}
Sep 09 17:30:21.929060 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.929038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-d42fs" event={"ID":"0a30b228-1b96-47d9-98a6-e70dcda57b05","Type":"ContainerStarted","Data":"b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0"}
Sep 09 17:30:21.929137 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.929068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-d42fs" event={"ID":"0a30b228-1b96-47d9-98a6-e70dcda57b05","Type":"ContainerStarted","Data":"758e7813079421bcf7d4849f47eda4bae671341cbb03f72747cb14e7092f588d"}
Sep 09 17:30:21.929137 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.929108    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-d42fs" podUID="0a30b228-1b96-47d9-98a6-e70dcda57b05" containerName="authorino" containerID="cri-o://b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0" gracePeriod=30
Sep 09 17:30:21.943881 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.943835    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-744949959-jfbbs" podStartSLOduration=1.525216614 podStartE2EDuration="1.94381616s" totalImagesPullingTime="418.599546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:21.943625668 +0000 UTC m=+797.571778383" watchObservedRunningTime="2026-09-09 17:30:21.94381616 +0000 UTC m=+797.571968865"
Sep 09 17:30:21.960227 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.960178    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-d42fs" podStartSLOduration=1.288345981 podStartE2EDuration="1.960161557s" totalImagesPullingTime="671.815576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:21.957942026 +0000 UTC m=+797.586094729" watchObservedRunningTime="2026-09-09 17:30:21.960161557 +0000 UTC m=+797.588314259"
Sep 09 17:30:21.973841 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.973811    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f67s8"]
Sep 09 17:30:21.974011 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:21.973992    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-f67s8" podUID="d4f0a12a-ee15-479b-a640-1bd2d3a0a825" containerName="authorino" containerID="cri-o://43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9" gracePeriod=30
Sep 09 17:30:22.057966 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.057915    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f5d57569-wqzt9"]
Sep 09 17:30:22.060997 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.060978    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.063515 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.063497    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 17:30:22.068911 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.068886    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f5d57569-wqzt9"]
Sep 09 17:30:22.079278 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.079249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3332853f-6cdd-41c3-b99c-315df628d8a6-tls-cert\") pod \"authorino-f5d57569-wqzt9\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") " pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.079410 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.079288    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twgr\" (UniqueName: \"kubernetes.io/projected/3332853f-6cdd-41c3-b99c-315df628d8a6-kube-api-access-9twgr\") pod \"authorino-f5d57569-wqzt9\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") " pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.179907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.179836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3332853f-6cdd-41c3-b99c-315df628d8a6-tls-cert\") pod \"authorino-f5d57569-wqzt9\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") " pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.179907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.179882    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twgr\" (UniqueName: \"kubernetes.io/projected/3332853f-6cdd-41c3-b99c-315df628d8a6-kube-api-access-9twgr\") pod \"authorino-f5d57569-wqzt9\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") " pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.182537 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.182509    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3332853f-6cdd-41c3-b99c-315df628d8a6-tls-cert\") pod \"authorino-f5d57569-wqzt9\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") " pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.188176 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.188153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twgr\" (UniqueName: \"kubernetes.io/projected/3332853f-6cdd-41c3-b99c-315df628d8a6-kube-api-access-9twgr\") pod \"authorino-f5d57569-wqzt9\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") " pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.198917 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.198896    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-d42fs"
Sep 09 17:30:22.235262 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.235236    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f67s8"
Sep 09 17:30:22.280921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.280875    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2crjr\" (UniqueName: \"kubernetes.io/projected/0a30b228-1b96-47d9-98a6-e70dcda57b05-kube-api-access-2crjr\") pod \"0a30b228-1b96-47d9-98a6-e70dcda57b05\" (UID: \"0a30b228-1b96-47d9-98a6-e70dcda57b05\") "
Sep 09 17:30:22.282872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.282845    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a30b228-1b96-47d9-98a6-e70dcda57b05-kube-api-access-2crjr" (OuterVolumeSpecName: "kube-api-access-2crjr") pod "0a30b228-1b96-47d9-98a6-e70dcda57b05" (UID: "0a30b228-1b96-47d9-98a6-e70dcda57b05"). InnerVolumeSpecName "kube-api-access-2crjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:30:22.370408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.370375    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:30:22.381412 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.381387    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-httpb\" (UniqueName: \"kubernetes.io/projected/d4f0a12a-ee15-479b-a640-1bd2d3a0a825-kube-api-access-httpb\") pod \"d4f0a12a-ee15-479b-a640-1bd2d3a0a825\" (UID: \"d4f0a12a-ee15-479b-a640-1bd2d3a0a825\") "
Sep 09 17:30:22.381544 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.381531    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crjr\" (UniqueName: \"kubernetes.io/projected/0a30b228-1b96-47d9-98a6-e70dcda57b05-kube-api-access-2crjr\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:22.383240 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.383212    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f0a12a-ee15-479b-a640-1bd2d3a0a825-kube-api-access-httpb" (OuterVolumeSpecName: "kube-api-access-httpb") pod "d4f0a12a-ee15-479b-a640-1bd2d3a0a825" (UID: "d4f0a12a-ee15-479b-a640-1bd2d3a0a825"). InnerVolumeSpecName "kube-api-access-httpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:30:22.482520 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.482424    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-httpb\" (UniqueName: \"kubernetes.io/projected/d4f0a12a-ee15-479b-a640-1bd2d3a0a825-kube-api-access-httpb\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:22.647212 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.647186    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f5d57569-wqzt9"]
Sep 09 17:30:22.648801 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:30:22.648772    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3332853f_6cdd_41c3_b99c_315df628d8a6.slice/crio-0130962e62f1a100cc96b7b78859cf8a1e6c8af1a6e7c88b5e20c7b2affe035c WatchSource:0}: Error finding container 0130962e62f1a100cc96b7b78859cf8a1e6c8af1a6e7c88b5e20c7b2affe035c: Status 404 returned error can't find the container with id 0130962e62f1a100cc96b7b78859cf8a1e6c8af1a6e7c88b5e20c7b2affe035c
Sep 09 17:30:22.934353 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.934315    2558 generic.go:356] "Generic (PLEG): container finished" podID="0a30b228-1b96-47d9-98a6-e70dcda57b05" containerID="b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0" exitCode=0
Sep 09 17:30:22.934528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.934362    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-d42fs"
Sep 09 17:30:22.934528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.934401    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-d42fs" event={"ID":"0a30b228-1b96-47d9-98a6-e70dcda57b05","Type":"ContainerDied","Data":"b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0"}
Sep 09 17:30:22.934528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.934449    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-d42fs" event={"ID":"0a30b228-1b96-47d9-98a6-e70dcda57b05","Type":"ContainerDied","Data":"758e7813079421bcf7d4849f47eda4bae671341cbb03f72747cb14e7092f588d"}
Sep 09 17:30:22.934528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.934470    2558 scope.go:117] "RemoveContainer" containerID="b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0"
Sep 09 17:30:22.935723 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.935652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f5d57569-wqzt9" event={"ID":"3332853f-6cdd-41c3-b99c-315df628d8a6","Type":"ContainerStarted","Data":"0130962e62f1a100cc96b7b78859cf8a1e6c8af1a6e7c88b5e20c7b2affe035c"}
Sep 09 17:30:22.937165 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.937145    2558 generic.go:356] "Generic (PLEG): container finished" podID="d4f0a12a-ee15-479b-a640-1bd2d3a0a825" containerID="43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9" exitCode=0
Sep 09 17:30:22.937269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.937171    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-f67s8"
Sep 09 17:30:22.937269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.937202    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f67s8" event={"ID":"d4f0a12a-ee15-479b-a640-1bd2d3a0a825","Type":"ContainerDied","Data":"43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9"}
Sep 09 17:30:22.937269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.937223    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-f67s8" event={"ID":"d4f0a12a-ee15-479b-a640-1bd2d3a0a825","Type":"ContainerDied","Data":"d75417eb22285160a9896bf6ab6fbf5e3ac8d438e0931897895451ab7dadd387"}
Sep 09 17:30:22.943139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.943092    2558 scope.go:117] "RemoveContainer" containerID="b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0"
Sep 09 17:30:22.943380 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:30:22.943362    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0\": container with ID starting with b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0 not found: ID does not exist" containerID="b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0"
Sep 09 17:30:22.943432 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.943388    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0"} err="failed to get container status \"b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0\": rpc error: code = NotFound desc = could not find container \"b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0\": container with ID starting with b8e45702af08efb2acfb993576625027ddbf70b4eaa30e52189c486b007fa4a0 not found: ID does not exist"
Sep 09 17:30:22.943432 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.943405    2558 scope.go:117] "RemoveContainer" containerID="43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9"
Sep 09 17:30:22.950737 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.950715    2558 scope.go:117] "RemoveContainer" containerID="43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9"
Sep 09 17:30:22.951029 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:30:22.951011    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9\": container with ID starting with 43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9 not found: ID does not exist" containerID="43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9"
Sep 09 17:30:22.951092 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.951037    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9"} err="failed to get container status \"43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9\": rpc error: code = NotFound desc = could not find container \"43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9\": container with ID starting with 43d4f5196958036b8c3d11fd30ff581e58ab30e101a1c7522a604903bf58ece9 not found: ID does not exist"
Sep 09 17:30:22.964733 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.964703    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-d42fs"]
Sep 09 17:30:22.979640 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:22.979614    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-d42fs"]
Sep 09 17:30:23.002350 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:23.002325    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f67s8"]
Sep 09 17:30:23.006411 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:23.006389    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-f67s8"]
Sep 09 17:30:23.944380 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:23.944345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f5d57569-wqzt9" event={"ID":"3332853f-6cdd-41c3-b99c-315df628d8a6","Type":"ContainerStarted","Data":"8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d"}
Sep 09 17:30:23.971259 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:23.971201    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f5d57569-wqzt9" podStartSLOduration=1.435388007 podStartE2EDuration="1.971185531s" totalImagesPullingTime="535.797524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:23.969628687 +0000 UTC m=+799.597781403" watchObservedRunningTime="2026-09-09 17:30:23.971185531 +0000 UTC m=+799.599338233"
Sep 09 17:30:24.010419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.010376    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-744949959-jfbbs"]
Sep 09 17:30:24.010580 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.010546    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-744949959-jfbbs" podUID="a3e37c45-12fc-4e38-991c-8c43ca84bac9" containerName="authorino" containerID="cri-o://165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6" gracePeriod=30
Sep 09 17:30:24.261050 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.261025    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-744949959-jfbbs"
Sep 09 17:30:24.294466 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.294431    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcgt4\" (UniqueName: \"kubernetes.io/projected/a3e37c45-12fc-4e38-991c-8c43ca84bac9-kube-api-access-mcgt4\") pod \"a3e37c45-12fc-4e38-991c-8c43ca84bac9\" (UID: \"a3e37c45-12fc-4e38-991c-8c43ca84bac9\") "
Sep 09 17:30:24.296423 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.296398    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e37c45-12fc-4e38-991c-8c43ca84bac9-kube-api-access-mcgt4" (OuterVolumeSpecName: "kube-api-access-mcgt4") pod "a3e37c45-12fc-4e38-991c-8c43ca84bac9" (UID: "a3e37c45-12fc-4e38-991c-8c43ca84bac9"). InnerVolumeSpecName "kube-api-access-mcgt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:30:24.395847 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.395797    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcgt4\" (UniqueName: \"kubernetes.io/projected/a3e37c45-12fc-4e38-991c-8c43ca84bac9-kube-api-access-mcgt4\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:24.816199 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.816166    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a30b228-1b96-47d9-98a6-e70dcda57b05" path="/var/lib/kubelet/pods/0a30b228-1b96-47d9-98a6-e70dcda57b05/volumes"
Sep 09 17:30:24.816476 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.816464    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f0a12a-ee15-479b-a640-1bd2d3a0a825" path="/var/lib/kubelet/pods/d4f0a12a-ee15-479b-a640-1bd2d3a0a825/volumes"
Sep 09 17:30:24.951345 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.951312    2558 generic.go:356] "Generic (PLEG): container finished" podID="a3e37c45-12fc-4e38-991c-8c43ca84bac9" containerID="165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6" exitCode=0
Sep 09 17:30:24.951345 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.951348    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-744949959-jfbbs"
Sep 09 17:30:24.951574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.951387    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-744949959-jfbbs" event={"ID":"a3e37c45-12fc-4e38-991c-8c43ca84bac9","Type":"ContainerDied","Data":"165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6"}
Sep 09 17:30:24.951574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.951425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-744949959-jfbbs" event={"ID":"a3e37c45-12fc-4e38-991c-8c43ca84bac9","Type":"ContainerDied","Data":"0f00ae70deb9b27c32dda0b66c0ccd42eb1d3aa7d764c68fb5211a05a37b8de9"}
Sep 09 17:30:24.951574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.951448    2558 scope.go:117] "RemoveContainer" containerID="165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6"
Sep 09 17:30:24.959725 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.959709    2558 scope.go:117] "RemoveContainer" containerID="165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6"
Sep 09 17:30:24.960039 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:30:24.960017    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6\": container with ID starting with 165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6 not found: ID does not exist" containerID="165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6"
Sep 09 17:30:24.960141 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.960043    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6"} err="failed to get container status \"165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6\": rpc error: code = NotFound desc = could not find container \"165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6\": container with ID starting with 165cb7f4dd93cc8aa9ac2ed93d9fdff9eb6e082ab53e9204bbb3954fabcd3fd6 not found: ID does not exist"
Sep 09 17:30:24.967497 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.967470    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-744949959-jfbbs"]
Sep 09 17:30:24.972987 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:24.970021    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-744949959-jfbbs"]
Sep 09 17:30:26.772072 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772036    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57695cf7b5-sp8sp"]
Sep 09 17:30:26.772696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772673    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f0a12a-ee15-479b-a640-1bd2d3a0a825" containerName="authorino"
Sep 09 17:30:26.772773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772699    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f0a12a-ee15-479b-a640-1bd2d3a0a825" containerName="authorino"
Sep 09 17:30:26.772773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772723    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a30b228-1b96-47d9-98a6-e70dcda57b05" containerName="authorino"
Sep 09 17:30:26.772773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772731    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a30b228-1b96-47d9-98a6-e70dcda57b05" containerName="authorino"
Sep 09 17:30:26.772773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772741    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e37c45-12fc-4e38-991c-8c43ca84bac9" containerName="authorino"
Sep 09 17:30:26.772773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772749    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e37c45-12fc-4e38-991c-8c43ca84bac9" containerName="authorino"
Sep 09 17:30:26.773014 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772836    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f0a12a-ee15-479b-a640-1bd2d3a0a825" containerName="authorino"
Sep 09 17:30:26.773014 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772848    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a30b228-1b96-47d9-98a6-e70dcda57b05" containerName="authorino"
Sep 09 17:30:26.773014 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.772858    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e37c45-12fc-4e38-991c-8c43ca84bac9" containerName="authorino"
Sep 09 17:30:26.878443 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.878395    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57695cf7b5-sp8sp"]
Sep 09 17:30:26.878443 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.878426    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:26.883734 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.883706    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 17:30:26.884592 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.884566    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e37c45-12fc-4e38-991c-8c43ca84bac9" path="/var/lib/kubelet/pods/a3e37c45-12fc-4e38-991c-8c43ca84bac9/volumes"
Sep 09 17:30:26.885097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.885081    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fh5cx\""
Sep 09 17:30:26.885097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.885087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 17:30:26.885233 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.885080    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 17:30:26.914066 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.914034    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcfm\" (UniqueName: \"kubernetes.io/projected/44bae4d7-75e5-4cef-8e61-51f781e715a0-kube-api-access-qjcfm\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:26.914210 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.914080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-metrics-certs\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:26.914210 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.914133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-webhook-cert\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:26.993285 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:26.993248    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5474dd89c7-r87sq"]
Sep 09 17:30:27.015318 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.015285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcfm\" (UniqueName: \"kubernetes.io/projected/44bae4d7-75e5-4cef-8e61-51f781e715a0-kube-api-access-qjcfm\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:27.015498 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.015346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-metrics-certs\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:27.015498 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.015377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-webhook-cert\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:27.017903 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.017869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-webhook-cert\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:27.017903 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.017893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-metrics-certs\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:27.024016 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.023918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcfm\" (UniqueName: \"kubernetes.io/projected/44bae4d7-75e5-4cef-8e61-51f781e715a0-kube-api-access-qjcfm\") pod \"maas-controller-57695cf7b5-sp8sp\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") " pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:27.037059 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.037027    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-r87sq"]
Sep 09 17:30:27.037194 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.037137    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.116304 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.116222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-webhook-cert\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.116304 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.116263    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5st\" (UniqueName: \"kubernetes.io/projected/03dfc5ea-523f-497c-a704-10c15bb768ba-kube-api-access-8f5st\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.116304 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.116290    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-metrics-certs\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.189268 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.189228    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:27.217204 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.217172    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-webhook-cert\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.217333 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.217218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5st\" (UniqueName: \"kubernetes.io/projected/03dfc5ea-523f-497c-a704-10c15bb768ba-kube-api-access-8f5st\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.217333 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.217245    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-metrics-certs\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.219609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.219579    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-metrics-certs\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.219738 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.219616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-webhook-cert\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.225237 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.225214    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5st\" (UniqueName: \"kubernetes.io/projected/03dfc5ea-523f-497c-a704-10c15bb768ba-kube-api-access-8f5st\") pod \"maas-controller-5474dd89c7-r87sq\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") " pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.347554 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.347511    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:27.494843 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.494818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57695cf7b5-sp8sp"]
Sep 09 17:30:27.496520 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:30:27.496481    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bae4d7_75e5_4cef_8e61_51f781e715a0.slice/crio-ded4f80d450e2036c57e53dedf4de88bc89703acf624813a2bf188340f4e33e7 WatchSource:0}: Error finding container ded4f80d450e2036c57e53dedf4de88bc89703acf624813a2bf188340f4e33e7: Status 404 returned error can't find the container with id ded4f80d450e2036c57e53dedf4de88bc89703acf624813a2bf188340f4e33e7
Sep 09 17:30:27.650575 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.650549    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-r87sq"]
Sep 09 17:30:27.651858 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:30:27.651819    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dfc5ea_523f_497c_a704_10c15bb768ba.slice/crio-5755a23d6ed375107732e3bc1eeb3e4839e4ee161cfda9feff31aba4dfcf43e2 WatchSource:0}: Error finding container 5755a23d6ed375107732e3bc1eeb3e4839e4ee161cfda9feff31aba4dfcf43e2: Status 404 returned error can't find the container with id 5755a23d6ed375107732e3bc1eeb3e4839e4ee161cfda9feff31aba4dfcf43e2
Sep 09 17:30:27.967314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.967279    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-r87sq" event={"ID":"03dfc5ea-523f-497c-a704-10c15bb768ba","Type":"ContainerStarted","Data":"5755a23d6ed375107732e3bc1eeb3e4839e4ee161cfda9feff31aba4dfcf43e2"}
Sep 09 17:30:27.968437 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:27.968413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57695cf7b5-sp8sp" event={"ID":"44bae4d7-75e5-4cef-8e61-51f781e715a0","Type":"ContainerStarted","Data":"ded4f80d450e2036c57e53dedf4de88bc89703acf624813a2bf188340f4e33e7"}
Sep 09 17:30:30.990644 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:30.990607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-r87sq" event={"ID":"03dfc5ea-523f-497c-a704-10c15bb768ba","Type":"ContainerStarted","Data":"656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e"}
Sep 09 17:30:30.991246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:30.990760    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:30.992247 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:30.992217    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57695cf7b5-sp8sp" event={"ID":"44bae4d7-75e5-4cef-8e61-51f781e715a0","Type":"ContainerStarted","Data":"e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18"}
Sep 09 17:30:30.992368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:30.992271    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:31.010685 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:31.010629    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5474dd89c7-r87sq" podStartSLOduration=2.409237316 podStartE2EDuration="5.010613184s" totalImagesPullingTime="2.601375868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:31.008830828 +0000 UTC m=+806.636983568" watchObservedRunningTime="2026-09-09 17:30:31.010613184 +0000 UTC m=+806.638765886"
Sep 09 17:30:31.028882 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:31.028819    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57695cf7b5-sp8sp" podStartSLOduration=2.262294784 podStartE2EDuration="5.028800233s" totalImagesPullingTime="2.766505449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:30:31.027328882 +0000 UTC m=+806.655481586" watchObservedRunningTime="2026-09-09 17:30:31.028800233 +0000 UTC m=+806.656952963"
Sep 09 17:30:42.000096 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.000066    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:42.000478 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.000142    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:30:42.059857 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.059821    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57695cf7b5-sp8sp"]
Sep 09 17:30:42.060069 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.060021    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57695cf7b5-sp8sp" podUID="44bae4d7-75e5-4cef-8e61-51f781e715a0" containerName="manager" containerID="cri-o://e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18" gracePeriod=30
Sep 09 17:30:42.300083 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.300058    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:42.336021 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.335989    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjcfm\" (UniqueName: \"kubernetes.io/projected/44bae4d7-75e5-4cef-8e61-51f781e715a0-kube-api-access-qjcfm\") pod \"44bae4d7-75e5-4cef-8e61-51f781e715a0\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") "
Sep 09 17:30:42.336190 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.336039    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-metrics-certs\") pod \"44bae4d7-75e5-4cef-8e61-51f781e715a0\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") "
Sep 09 17:30:42.336190 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.336119    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-webhook-cert\") pod \"44bae4d7-75e5-4cef-8e61-51f781e715a0\" (UID: \"44bae4d7-75e5-4cef-8e61-51f781e715a0\") "
Sep 09 17:30:42.338247 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.338213    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bae4d7-75e5-4cef-8e61-51f781e715a0-kube-api-access-qjcfm" (OuterVolumeSpecName: "kube-api-access-qjcfm") pod "44bae4d7-75e5-4cef-8e61-51f781e715a0" (UID: "44bae4d7-75e5-4cef-8e61-51f781e715a0"). InnerVolumeSpecName "kube-api-access-qjcfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:30:42.338247 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.338229    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "44bae4d7-75e5-4cef-8e61-51f781e715a0" (UID: "44bae4d7-75e5-4cef-8e61-51f781e715a0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:30:42.338388 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.338248    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "44bae4d7-75e5-4cef-8e61-51f781e715a0" (UID: "44bae4d7-75e5-4cef-8e61-51f781e715a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:30:42.436839 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.436802    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjcfm\" (UniqueName: \"kubernetes.io/projected/44bae4d7-75e5-4cef-8e61-51f781e715a0-kube-api-access-qjcfm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:42.436839 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.436833    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-metrics-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:42.436839 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:42.436842    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44bae4d7-75e5-4cef-8e61-51f781e715a0-webhook-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:30:43.054488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.054449    2558 generic.go:356] "Generic (PLEG): container finished" podID="44bae4d7-75e5-4cef-8e61-51f781e715a0" containerID="e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18" exitCode=0
Sep 09 17:30:43.054921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.054493    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57695cf7b5-sp8sp"
Sep 09 17:30:43.054921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.054533    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57695cf7b5-sp8sp" event={"ID":"44bae4d7-75e5-4cef-8e61-51f781e715a0","Type":"ContainerDied","Data":"e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18"}
Sep 09 17:30:43.054921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.054587    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57695cf7b5-sp8sp" event={"ID":"44bae4d7-75e5-4cef-8e61-51f781e715a0","Type":"ContainerDied","Data":"ded4f80d450e2036c57e53dedf4de88bc89703acf624813a2bf188340f4e33e7"}
Sep 09 17:30:43.054921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.054608    2558 scope.go:117] "RemoveContainer" containerID="e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18"
Sep 09 17:30:43.063122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.062917    2558 scope.go:117] "RemoveContainer" containerID="e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18"
Sep 09 17:30:43.063272 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:30:43.063253    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18\": container with ID starting with e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18 not found: ID does not exist" containerID="e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18"
Sep 09 17:30:43.063320 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.063281    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18"} err="failed to get container status \"e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18\": rpc error: code = NotFound desc = could not find container \"e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18\": container with ID starting with e78463d2d963f6885db17897f0077349f115f8c67e2b2f8ee73483b319f53a18 not found: ID does not exist"
Sep 09 17:30:43.076031 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.072897    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57695cf7b5-sp8sp"]
Sep 09 17:30:43.078383 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:43.078359    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57695cf7b5-sp8sp"]
Sep 09 17:30:44.817421 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:30:44.817387    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44bae4d7-75e5-4cef-8e61-51f781e715a0" path="/var/lib/kubelet/pods/44bae4d7-75e5-4cef-8e61-51f781e715a0/volumes"
Sep 09 17:31:04.562366 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.562326    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"]
Sep 09 17:31:04.562823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.562623    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bae4d7-75e5-4cef-8e61-51f781e715a0" containerName="manager"
Sep 09 17:31:04.562823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.562635    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bae4d7-75e5-4cef-8e61-51f781e715a0" containerName="manager"
Sep 09 17:31:04.562823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.562697    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="44bae4d7-75e5-4cef-8e61-51f781e715a0" containerName="manager"
Sep 09 17:31:04.565526 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.565505    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.568657 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.568620    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6f59k\""
Sep 09 17:31:04.569034 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.569012    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 17:31:04.575437 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.575409    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"]
Sep 09 17:31:04.605496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.605456    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-95zgv"]
Sep 09 17:31:04.608590 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.608566    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:04.619498 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.619467    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-95zgv"]
Sep 09 17:31:04.709920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.709878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5svk\" (UniqueName: \"kubernetes.io/projected/d2f6aefe-ce42-414c-a1ce-fc534c57a331-kube-api-access-g5svk\") pod \"payload-processing-5b54b7fbb7-95zgv\" (UID: \"d2f6aefe-ce42-414c-a1ce-fc534c57a331\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:04.710148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.709980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6aefe-ce42-414c-a1ce-fc534c57a331-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-95zgv\" (UID: \"d2f6aefe-ce42-414c-a1ce-fc534c57a331\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:04.710148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.710018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1e33b-0d25-4a1e-8aa2-dce604fda858-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4gqc5\" (UID: \"22e1e33b-0d25-4a1e-8aa2-dce604fda858\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.710148 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.710052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29l9c\" (UniqueName: \"kubernetes.io/projected/22e1e33b-0d25-4a1e-8aa2-dce604fda858-kube-api-access-29l9c\") pod \"payload-pre-processing-59fbc47bf8-4gqc5\" (UID: \"22e1e33b-0d25-4a1e-8aa2-dce604fda858\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.810577 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.810527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5svk\" (UniqueName: \"kubernetes.io/projected/d2f6aefe-ce42-414c-a1ce-fc534c57a331-kube-api-access-g5svk\") pod \"payload-processing-5b54b7fbb7-95zgv\" (UID: \"d2f6aefe-ce42-414c-a1ce-fc534c57a331\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:04.810823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.810607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6aefe-ce42-414c-a1ce-fc534c57a331-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-95zgv\" (UID: \"d2f6aefe-ce42-414c-a1ce-fc534c57a331\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:04.810823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.810640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1e33b-0d25-4a1e-8aa2-dce604fda858-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4gqc5\" (UID: \"22e1e33b-0d25-4a1e-8aa2-dce604fda858\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.810823 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.810673    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29l9c\" (UniqueName: \"kubernetes.io/projected/22e1e33b-0d25-4a1e-8aa2-dce604fda858-kube-api-access-29l9c\") pod \"payload-pre-processing-59fbc47bf8-4gqc5\" (UID: \"22e1e33b-0d25-4a1e-8aa2-dce604fda858\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.813508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.813428    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 17:31:04.818580 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.818552    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29l9c\" (UniqueName: \"kubernetes.io/projected/22e1e33b-0d25-4a1e-8aa2-dce604fda858-kube-api-access-29l9c\") pod \"payload-pre-processing-59fbc47bf8-4gqc5\" (UID: \"22e1e33b-0d25-4a1e-8aa2-dce604fda858\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.818707 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.818558    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5svk\" (UniqueName: \"kubernetes.io/projected/d2f6aefe-ce42-414c-a1ce-fc534c57a331-kube-api-access-g5svk\") pod \"payload-processing-5b54b7fbb7-95zgv\" (UID: \"d2f6aefe-ce42-414c-a1ce-fc534c57a331\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:04.821738 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.821715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6aefe-ce42-414c-a1ce-fc534c57a331-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-95zgv\" (UID: \"d2f6aefe-ce42-414c-a1ce-fc534c57a331\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:04.822222 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.822205    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1e33b-0d25-4a1e-8aa2-dce604fda858-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4gqc5\" (UID: \"22e1e33b-0d25-4a1e-8aa2-dce604fda858\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.881288 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.881258    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6f59k\""
Sep 09 17:31:04.888746 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.888712    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:04.918294 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:04.918262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:05.174726 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:05.174704    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"]
Sep 09 17:31:05.176668 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:31:05.176637    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e1e33b_0d25_4a1e_8aa2_dce604fda858.slice/crio-c0a08d6bdac88089924493b2520be00fb763ecec1f23a551bba223f82c0a8aee WatchSource:0}: Error finding container c0a08d6bdac88089924493b2520be00fb763ecec1f23a551bba223f82c0a8aee: Status 404 returned error can't find the container with id c0a08d6bdac88089924493b2520be00fb763ecec1f23a551bba223f82c0a8aee
Sep 09 17:31:05.178532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:05.178517    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 17:31:05.204426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:05.204377    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-95zgv"]
Sep 09 17:31:05.206534 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:31:05.206508    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f6aefe_ce42_414c_a1ce_fc534c57a331.slice/crio-9d279bf49e5ddf2aa6b69915aae5a05a6b6c383c4f8b8ff8ec74be33dd904855 WatchSource:0}: Error finding container 9d279bf49e5ddf2aa6b69915aae5a05a6b6c383c4f8b8ff8ec74be33dd904855: Status 404 returned error can't find the container with id 9d279bf49e5ddf2aa6b69915aae5a05a6b6c383c4f8b8ff8ec74be33dd904855
Sep 09 17:31:06.168687 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:06.168593    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv" event={"ID":"d2f6aefe-ce42-414c-a1ce-fc534c57a331","Type":"ContainerStarted","Data":"9d279bf49e5ddf2aa6b69915aae5a05a6b6c383c4f8b8ff8ec74be33dd904855"}
Sep 09 17:31:06.169856 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:06.169828    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5" event={"ID":"22e1e33b-0d25-4a1e-8aa2-dce604fda858","Type":"ContainerStarted","Data":"c0a08d6bdac88089924493b2520be00fb763ecec1f23a551bba223f82c0a8aee"}
Sep 09 17:31:08.182901 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:08.182864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5" event={"ID":"22e1e33b-0d25-4a1e-8aa2-dce604fda858","Type":"ContainerStarted","Data":"9c38b547879e1c83ab0556d26fb8eeb25d15591034c2d9cbe6dcdea912f4860a"}
Sep 09 17:31:08.183381 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:08.182949    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:08.184502 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:08.184478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv" event={"ID":"d2f6aefe-ce42-414c-a1ce-fc534c57a331","Type":"ContainerStarted","Data":"69273efc33cd92e74e8deeff95a77b869d3220d5a8f9da619545db73a39bdaa9"}
Sep 09 17:31:08.184606 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:08.184536    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:31:08.198237 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:08.198183    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5" podStartSLOduration=1.9348842469999998 podStartE2EDuration="4.198167298s" totalImagesPullingTime="2.263283051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:31:08.19797853 +0000 UTC m=+843.826131229" watchObservedRunningTime="2026-09-09 17:31:08.198167298 +0000 UTC m=+843.826320004"
Sep 09 17:31:08.213420 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:08.213366    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv" podStartSLOduration=1.97390839 podStartE2EDuration="4.213347406s" totalImagesPullingTime="2.239439016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:31:08.212753319 +0000 UTC m=+843.840906022" watchObservedRunningTime="2026-09-09 17:31:08.213347406 +0000 UTC m=+843.841500108"
Sep 09 17:31:19.190718 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:19.190683    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4gqc5"
Sep 09 17:31:19.191307 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:31:19.190847    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-95zgv"
Sep 09 17:32:02.189887 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.189809    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"]
Sep 09 17:32:02.193155 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.193131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.196501 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.196481    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 17:32:02.197774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.197755    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7dhnx\""
Sep 09 17:32:02.198380 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.198254    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 17:32:02.198380 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.198348    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 17:32:02.205929 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.205902    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"]
Sep 09 17:32:02.270143 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.270110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.270274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.270150    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e724880-5662-4425-899e-4e9d2e69615a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.270274 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.270169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.270344 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.270265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.270344 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.270309    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5w4q\" (UniqueName: \"kubernetes.io/projected/1e724880-5662-4425-899e-4e9d2e69615a-kube-api-access-q5w4q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.270410 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.270361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371020 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.370981    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4q\" (UniqueName: \"kubernetes.io/projected/1e724880-5662-4425-899e-4e9d2e69615a-kube-api-access-q5w4q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371061    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371087    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371124    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e724880-5662-4425-899e-4e9d2e69615a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371484 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371459    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.371608 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.371582    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.373426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.373405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e724880-5662-4425-899e-4e9d2e69615a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.373651 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.373631    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e724880-5662-4425-899e-4e9d2e69615a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.382577 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.382553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5w4q\" (UniqueName: \"kubernetes.io/projected/1e724880-5662-4425-899e-4e9d2e69615a-kube-api-access-q5w4q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq\" (UID: \"1e724880-5662-4425-899e-4e9d2e69615a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.522092 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.522002    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:02.790983 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:02.790945    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"]
Sep 09 17:32:02.793311 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:32:02.793286    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e724880_5662_4425_899e_4e9d2e69615a.slice/crio-53203ad6f3dbd95192ce992a4d908f9886033c3cbfaa1d3c9f2299abd19d6493 WatchSource:0}: Error finding container 53203ad6f3dbd95192ce992a4d908f9886033c3cbfaa1d3c9f2299abd19d6493: Status 404 returned error can't find the container with id 53203ad6f3dbd95192ce992a4d908f9886033c3cbfaa1d3c9f2299abd19d6493
Sep 09 17:32:03.449334 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:03.449288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq" event={"ID":"1e724880-5662-4425-899e-4e9d2e69615a","Type":"ContainerStarted","Data":"53203ad6f3dbd95192ce992a4d908f9886033c3cbfaa1d3c9f2299abd19d6493"}
Sep 09 17:32:04.456549 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:04.456505    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq" event={"ID":"1e724880-5662-4425-899e-4e9d2e69615a","Type":"ContainerStarted","Data":"80c35ae64f3a71e1dd581fbc375073a667941b3464d5b0ae2f8791a734107182"}
Sep 09 17:32:04.476127 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:04.476071    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq" podStartSLOduration=1.359358652 podStartE2EDuration="2.476051854s" totalImagesPullingTime="1.116693202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:32:04.475161402 +0000 UTC m=+900.103314105" watchObservedRunningTime="2026-09-09 17:32:04.476051854 +0000 UTC m=+900.104204557"
Sep 09 17:32:04.760425 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:04.760346    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:32:04.760425 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:04.760384    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:32:12.522780 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:12.522744    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:12.525369 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:12.522812    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:12.535332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:12.535309    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:13.286781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.286747    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"]
Sep 09 17:32:13.289799 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.289777    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.292367 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.292342    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 17:32:13.300524 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.300501    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"]
Sep 09 17:32:13.365903 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.365871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773972-ff4a-4f42-a60d-115371b3349d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.366089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.365911    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.366089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.365944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.366089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.365991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gr7\" (UniqueName: \"kubernetes.io/projected/ad773972-ff4a-4f42-a60d-115371b3349d-kube-api-access-84gr7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.366089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.366030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.366089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.366046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.466446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.466405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773972-ff4a-4f42-a60d-115371b3349d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.466446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.466452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.466696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.466572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.466696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.466606    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gr7\" (UniqueName: \"kubernetes.io/projected/ad773972-ff4a-4f42-a60d-115371b3349d-kube-api-access-84gr7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.466696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.466662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.466696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.466680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.467007 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.466984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.467110 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.467058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.467110 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.467098    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.468793 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.468775    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773972-ff4a-4f42-a60d-115371b3349d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.469182 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.469163    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773972-ff4a-4f42-a60d-115371b3349d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.474314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.474291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gr7\" (UniqueName: \"kubernetes.io/projected/ad773972-ff4a-4f42-a60d-115371b3349d-kube-api-access-84gr7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hwrdx\" (UID: \"ad773972-ff4a-4f42-a60d-115371b3349d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.509191 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.509164    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq"
Sep 09 17:32:13.601522 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.601433    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:13.873479 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:13.873450    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"]
Sep 09 17:32:13.874676 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:32:13.874650    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad773972_ff4a_4f42_a60d_115371b3349d.slice/crio-e83480a365ce2f4c0dc1f9bee3d7e62d1667e31a33b97e904979010a472044f6 WatchSource:0}: Error finding container e83480a365ce2f4c0dc1f9bee3d7e62d1667e31a33b97e904979010a472044f6: Status 404 returned error can't find the container with id e83480a365ce2f4c0dc1f9bee3d7e62d1667e31a33b97e904979010a472044f6
Sep 09 17:32:14.505034 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:14.505001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx" event={"ID":"ad773972-ff4a-4f42-a60d-115371b3349d","Type":"ContainerStarted","Data":"6d06e33cf8c77dcfc9d85beb7b27ca4b9b99a29fd8b55bf0f9b1c94c2deaa947"}
Sep 09 17:32:14.505220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:14.505058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx" event={"ID":"ad773972-ff4a-4f42-a60d-115371b3349d","Type":"ContainerStarted","Data":"e83480a365ce2f4c0dc1f9bee3d7e62d1667e31a33b97e904979010a472044f6"}
Sep 09 17:32:14.523665 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:14.523613    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx" podStartSLOduration=1.296487197 podStartE2EDuration="1.523598466s" totalImagesPullingTime="227.111269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:32:14.523022138 +0000 UTC m=+910.151174839" watchObservedRunningTime="2026-09-09 17:32:14.523598466 +0000 UTC m=+910.151751183"
Sep 09 17:32:20.582383 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.582347    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"]
Sep 09 17:32:20.585879 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.585858    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.588695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.588671    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 17:32:20.596080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.596056    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"]
Sep 09 17:32:20.727249 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.727207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.727249 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.727254    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.727446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.727334    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.727446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.727376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1f7675-1366-4c07-976e-18267b0542c9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.727446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.727395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nhjj\" (UniqueName: \"kubernetes.io/projected/ea1f7675-1366-4c07-976e-18267b0542c9-kube-api-access-7nhjj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.727546 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.727447    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.827810 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.827776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.827810 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.827815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1f7675-1366-4c07-976e-18267b0542c9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.828063 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.827831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nhjj\" (UniqueName: \"kubernetes.io/projected/ea1f7675-1366-4c07-976e-18267b0542c9-kube-api-access-7nhjj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.828063 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.827879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.828063 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.827916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.828063 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.827976    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.828266 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.828242    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.828331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.828270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.828331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.828323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.830134 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.830114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1f7675-1366-4c07-976e-18267b0542c9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.830368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.830350    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1f7675-1366-4c07-976e-18267b0542c9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.837442 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.837390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nhjj\" (UniqueName: \"kubernetes.io/projected/ea1f7675-1366-4c07-976e-18267b0542c9-kube-api-access-7nhjj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn\" (UID: \"ea1f7675-1366-4c07-976e-18267b0542c9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:20.897330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:20.897293    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:21.169061 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:21.169033    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"]
Sep 09 17:32:21.170828 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:32:21.170801    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea1f7675_1366_4c07_976e_18267b0542c9.slice/crio-4e6a5ba9bcd7cf9d344d5e15e3e94d631e1204eeeb725370486e8b3b11f169c5 WatchSource:0}: Error finding container 4e6a5ba9bcd7cf9d344d5e15e3e94d631e1204eeeb725370486e8b3b11f169c5: Status 404 returned error can't find the container with id 4e6a5ba9bcd7cf9d344d5e15e3e94d631e1204eeeb725370486e8b3b11f169c5
Sep 09 17:32:21.542339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:21.542301    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn" event={"ID":"ea1f7675-1366-4c07-976e-18267b0542c9","Type":"ContainerStarted","Data":"e24528b3df61a5f08fcae73ada83099f81c38a9fb4a2720338d8f382a96ba130"}
Sep 09 17:32:21.542339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:21.542345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn" event={"ID":"ea1f7675-1366-4c07-976e-18267b0542c9","Type":"ContainerStarted","Data":"4e6a5ba9bcd7cf9d344d5e15e3e94d631e1204eeeb725370486e8b3b11f169c5"}
Sep 09 17:32:21.559551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:21.559496    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn" podStartSLOduration=1.371136701 podStartE2EDuration="1.559478704s" totalImagesPullingTime="188.342003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:32:21.557238166 +0000 UTC m=+917.185390869" watchObservedRunningTime="2026-09-09 17:32:21.559478704 +0000 UTC m=+917.187631406"
Sep 09 17:32:23.602166 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:23.602123    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:23.602568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:23.602192    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:23.618525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:23.618483    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:24.568368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:24.568332    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hwrdx"
Sep 09 17:32:30.897872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:30.897832    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:30.898311 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:30.897884    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:30.910229 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:30.910202    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:32:31.603112 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:32:31.603060    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn"
Sep 09 17:33:40.663286 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.663208    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8db654b6d-zpcnv"]
Sep 09 17:33:40.666574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.666557    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:40.674490 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.674454    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8db654b6d-zpcnv"]
Sep 09 17:33:40.796028 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.795986    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62c85067-f231-4236-9c39-ce99c514a88d-tls-cert\") pod \"authorino-8db654b6d-zpcnv\" (UID: \"62c85067-f231-4236-9c39-ce99c514a88d\") " pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:40.796201 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.796066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzn7\" (UniqueName: \"kubernetes.io/projected/62c85067-f231-4236-9c39-ce99c514a88d-kube-api-access-2dzn7\") pod \"authorino-8db654b6d-zpcnv\" (UID: \"62c85067-f231-4236-9c39-ce99c514a88d\") " pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:40.897527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.897482    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62c85067-f231-4236-9c39-ce99c514a88d-tls-cert\") pod \"authorino-8db654b6d-zpcnv\" (UID: \"62c85067-f231-4236-9c39-ce99c514a88d\") " pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:40.897705 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.897554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzn7\" (UniqueName: \"kubernetes.io/projected/62c85067-f231-4236-9c39-ce99c514a88d-kube-api-access-2dzn7\") pod \"authorino-8db654b6d-zpcnv\" (UID: \"62c85067-f231-4236-9c39-ce99c514a88d\") " pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:40.900099 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.900073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62c85067-f231-4236-9c39-ce99c514a88d-tls-cert\") pod \"authorino-8db654b6d-zpcnv\" (UID: \"62c85067-f231-4236-9c39-ce99c514a88d\") " pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:40.906479 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.906458    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzn7\" (UniqueName: \"kubernetes.io/projected/62c85067-f231-4236-9c39-ce99c514a88d-kube-api-access-2dzn7\") pod \"authorino-8db654b6d-zpcnv\" (UID: \"62c85067-f231-4236-9c39-ce99c514a88d\") " pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:40.979393 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:40.979348    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8db654b6d-zpcnv"
Sep 09 17:33:41.249425 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:41.249394    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8db654b6d-zpcnv"]
Sep 09 17:33:41.251099 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:33:41.251041    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62c85067_f231_4236_9c39_ce99c514a88d.slice/crio-5ea099a9f8f63f359498ed1acf3fbedc9195115218150554b153e52215f13e54 WatchSource:0}: Error finding container 5ea099a9f8f63f359498ed1acf3fbedc9195115218150554b153e52215f13e54: Status 404 returned error can't find the container with id 5ea099a9f8f63f359498ed1acf3fbedc9195115218150554b153e52215f13e54
Sep 09 17:33:41.937155 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:41.937114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8db654b6d-zpcnv" event={"ID":"62c85067-f231-4236-9c39-ce99c514a88d","Type":"ContainerStarted","Data":"5ea099a9f8f63f359498ed1acf3fbedc9195115218150554b153e52215f13e54"}
Sep 09 17:33:42.946342 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:42.946292    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8db654b6d-zpcnv" event={"ID":"62c85067-f231-4236-9c39-ce99c514a88d","Type":"ContainerStarted","Data":"09b143ad577937709a9ede75e7a73725a40ba10a207621f28bbefd3896e10d02"}
Sep 09 17:33:42.975144 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:42.975066    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8db654b6d-zpcnv" podStartSLOduration=2.050574009 podStartE2EDuration="2.97504771s" totalImagesPullingTime="924.473701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:33:42.974067486 +0000 UTC m=+998.602220189" watchObservedRunningTime="2026-09-09 17:33:42.97504771 +0000 UTC m=+998.603200405"
Sep 09 17:33:43.016334 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.015975    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f5d57569-wqzt9"]
Sep 09 17:33:43.016334 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.016185    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-f5d57569-wqzt9" podUID="3332853f-6cdd-41c3-b99c-315df628d8a6" containerName="authorino" containerID="cri-o://8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d" gracePeriod=30
Sep 09 17:33:43.258812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.258784    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:33:43.320500 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.320458    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9twgr\" (UniqueName: \"kubernetes.io/projected/3332853f-6cdd-41c3-b99c-315df628d8a6-kube-api-access-9twgr\") pod \"3332853f-6cdd-41c3-b99c-315df628d8a6\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") "
Sep 09 17:33:43.320698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.320636    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3332853f-6cdd-41c3-b99c-315df628d8a6-tls-cert\") pod \"3332853f-6cdd-41c3-b99c-315df628d8a6\" (UID: \"3332853f-6cdd-41c3-b99c-315df628d8a6\") "
Sep 09 17:33:43.323227 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.323187    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3332853f-6cdd-41c3-b99c-315df628d8a6-kube-api-access-9twgr" (OuterVolumeSpecName: "kube-api-access-9twgr") pod "3332853f-6cdd-41c3-b99c-315df628d8a6" (UID: "3332853f-6cdd-41c3-b99c-315df628d8a6"). InnerVolumeSpecName "kube-api-access-9twgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:33:43.331672 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.331640    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3332853f-6cdd-41c3-b99c-315df628d8a6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3332853f-6cdd-41c3-b99c-315df628d8a6" (UID: "3332853f-6cdd-41c3-b99c-315df628d8a6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:33:43.421939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.421902    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3332853f-6cdd-41c3-b99c-315df628d8a6-tls-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:33:43.421939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.421933    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9twgr\" (UniqueName: \"kubernetes.io/projected/3332853f-6cdd-41c3-b99c-315df628d8a6-kube-api-access-9twgr\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:33:43.953286 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.953251    2558 generic.go:356] "Generic (PLEG): container finished" podID="3332853f-6cdd-41c3-b99c-315df628d8a6" containerID="8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d" exitCode=0
Sep 09 17:33:43.953680 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.953313    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5d57569-wqzt9"
Sep 09 17:33:43.953680 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.953343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f5d57569-wqzt9" event={"ID":"3332853f-6cdd-41c3-b99c-315df628d8a6","Type":"ContainerDied","Data":"8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d"}
Sep 09 17:33:43.953680 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.953387    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f5d57569-wqzt9" event={"ID":"3332853f-6cdd-41c3-b99c-315df628d8a6","Type":"ContainerDied","Data":"0130962e62f1a100cc96b7b78859cf8a1e6c8af1a6e7c88b5e20c7b2affe035c"}
Sep 09 17:33:43.953680 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.953407    2558 scope.go:117] "RemoveContainer" containerID="8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d"
Sep 09 17:33:43.961893 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.961870    2558 scope.go:117] "RemoveContainer" containerID="8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d"
Sep 09 17:33:43.962219 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:33:43.962189    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d\": container with ID starting with 8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d not found: ID does not exist" containerID="8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d"
Sep 09 17:33:43.962324 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.962220    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d"} err="failed to get container status \"8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d\": rpc error: code = NotFound desc = could not find container \"8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d\": container with ID starting with 8a6327f6880d0012545bae78ddbaf37086ea6a3788c061243fe3152d4bfe8a2d not found: ID does not exist"
Sep 09 17:33:43.976894 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.976869    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f5d57569-wqzt9"]
Sep 09 17:33:43.980444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:43.980422    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f5d57569-wqzt9"]
Sep 09 17:33:44.821464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:33:44.821415    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3332853f-6cdd-41c3-b99c-315df628d8a6" path="/var/lib/kubelet/pods/3332853f-6cdd-41c3-b99c-315df628d8a6/volumes"
Sep 09 17:34:06.963753 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.963717    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"]
Sep 09 17:34:06.964161 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.964031    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3332853f-6cdd-41c3-b99c-315df628d8a6" containerName="authorino"
Sep 09 17:34:06.964161 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.964043    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3332853f-6cdd-41c3-b99c-315df628d8a6" containerName="authorino"
Sep 09 17:34:06.964161 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.964105    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3332853f-6cdd-41c3-b99c-315df628d8a6" containerName="authorino"
Sep 09 17:34:06.968211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.968191    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:06.970962 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.970922    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ea996c-dockercfg-9l2wb\""
Sep 09 17:34:06.971072 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.970930    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ea996c\""
Sep 09 17:34:06.977511 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:06.977490    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"]
Sep 09 17:34:07.026408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.026374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:07.026568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.026429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2xj\" (UniqueName: \"kubernetes.io/projected/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-kube-api-access-6b2xj\") pod \"payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:07.127792 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.127752    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:07.127992 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.127805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2xj\" (UniqueName: \"kubernetes.io/projected/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-kube-api-access-6b2xj\") pod \"payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:07.128489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.128468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:07.136070 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.136040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2xj\" (UniqueName: \"kubernetes.io/projected/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-kube-api-access-6b2xj\") pod \"payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:07.278647 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.278559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:07.550379 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:07.550352    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"]
Sep 09 17:34:07.551974 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:34:07.551932    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7aee36_49c8_4c6c_a7a3_538aa314ed8c.slice/crio-48bfab409cb8d9a88c07eedf80847367cf94b0239036a1ab92d62a43b78f9733 WatchSource:0}: Error finding container 48bfab409cb8d9a88c07eedf80847367cf94b0239036a1ab92d62a43b78f9733: Status 404 returned error can't find the container with id 48bfab409cb8d9a88c07eedf80847367cf94b0239036a1ab92d62a43b78f9733
Sep 09 17:34:08.072461 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:08.072426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q" event={"ID":"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c","Type":"ContainerStarted","Data":"c8baf1ebc71dd230f2afca24e4da173719cf32bde5075ba1aa9d9937a8cab1f7"}
Sep 09 17:34:08.072913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:08.072477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q" event={"ID":"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c","Type":"ContainerStarted","Data":"48bfab409cb8d9a88c07eedf80847367cf94b0239036a1ab92d62a43b78f9733"}
Sep 09 17:34:08.072913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:08.072510    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:08.092314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:08.092240    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q" podStartSLOduration=2.092217027 podStartE2EDuration="2.092217027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:08.090097765 +0000 UTC m=+1023.718250469" watchObservedRunningTime="2026-09-09 17:34:08.092217027 +0000 UTC m=+1023.720369734"
Sep 09 17:34:14.146347 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.146305    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"]
Sep 09 17:34:14.149811 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.149791    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.152373 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.152353    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-399f043b\""
Sep 09 17:34:14.152479 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.152392    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-399f043b-dockercfg-mjs6h\""
Sep 09 17:34:14.158342 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.158322    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"]
Sep 09 17:34:14.291337 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.291299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzjq\" (UniqueName: \"kubernetes.io/projected/7743bc10-cb19-47c9-b07b-625f5373e4e3-kube-api-access-cmzjq\") pod \"payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.291536 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.291362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7743bc10-cb19-47c9-b07b-625f5373e4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.392165 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.392124    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7743bc10-cb19-47c9-b07b-625f5373e4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.392332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.392239    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzjq\" (UniqueName: \"kubernetes.io/projected/7743bc10-cb19-47c9-b07b-625f5373e4e3-kube-api-access-cmzjq\") pod \"payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.392907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.392886    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7743bc10-cb19-47c9-b07b-625f5373e4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.400088 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.400028    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzjq\" (UniqueName: \"kubernetes.io/projected/7743bc10-cb19-47c9-b07b-625f5373e4e3-kube-api-access-cmzjq\") pod \"payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.460545 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.460503    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:14.731187 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:14.731155    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"]
Sep 09 17:34:14.733549 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:34:14.733521    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7743bc10_cb19_47c9_b07b_625f5373e4e3.slice/crio-83a361aced16cdacb301fdad3ea89e0dd805ff492245080246dbe206257e2fe4 WatchSource:0}: Error finding container 83a361aced16cdacb301fdad3ea89e0dd805ff492245080246dbe206257e2fe4: Status 404 returned error can't find the container with id 83a361aced16cdacb301fdad3ea89e0dd805ff492245080246dbe206257e2fe4
Sep 09 17:34:15.113080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:15.112976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq" event={"ID":"7743bc10-cb19-47c9-b07b-625f5373e4e3","Type":"ContainerStarted","Data":"d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec"}
Sep 09 17:34:15.113080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:15.113039    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq" event={"ID":"7743bc10-cb19-47c9-b07b-625f5373e4e3","Type":"ContainerStarted","Data":"83a361aced16cdacb301fdad3ea89e0dd805ff492245080246dbe206257e2fe4"}
Sep 09 17:34:15.113305 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:15.113221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:15.131422 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:15.131376    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq" podStartSLOduration=1.131359385 podStartE2EDuration="1.131359385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:15.130302803 +0000 UTC m=+1030.758455507" watchObservedRunningTime="2026-09-09 17:34:15.131359385 +0000 UTC m=+1030.759512153"
Sep 09 17:34:16.744194 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.744160    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"]
Sep 09 17:34:16.747926 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.747904    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:16.750656 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.750631    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-37448c-dockercfg-ckb56\""
Sep 09 17:34:16.750794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.750663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-37448c\""
Sep 09 17:34:16.760058 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.760024    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"]
Sep 09 17:34:16.775644 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.775612    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"]
Sep 09 17:34:16.779209 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.779190    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:16.789532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.789506    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"]
Sep 09 17:34:16.816028 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.815998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60671309-799f-447d-a7ce-f4cf767c6547-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:16.816187 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.816038    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtlv\" (UniqueName: \"kubernetes.io/projected/60671309-799f-447d-a7ce-f4cf767c6547-kube-api-access-9wtlv\") pod \"payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:16.916725 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.916686    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60671309-799f-447d-a7ce-f4cf767c6547-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:16.916903 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.916742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtlv\" (UniqueName: \"kubernetes.io/projected/60671309-799f-447d-a7ce-f4cf767c6547-kube-api-access-9wtlv\") pod \"payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:16.916903 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.916791    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-plugins-config-volume\") pod \"payload-processing-e2e-autores-37448c-68dc44cbc9-j2789\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") " pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:16.917006 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.916945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7hj\" (UniqueName: \"kubernetes.io/projected/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-kube-api-access-tb7hj\") pod \"payload-processing-e2e-autores-37448c-68dc44cbc9-j2789\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") " pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:16.917456 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.917437    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60671309-799f-447d-a7ce-f4cf767c6547-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:16.924500 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:16.924474    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtlv\" (UniqueName: \"kubernetes.io/projected/60671309-799f-447d-a7ce-f4cf767c6547-kube-api-access-9wtlv\") pod \"payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:17.017935 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.017834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-plugins-config-volume\") pod \"payload-processing-e2e-autores-37448c-68dc44cbc9-j2789\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") " pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:17.017935 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.017904    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7hj\" (UniqueName: \"kubernetes.io/projected/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-kube-api-access-tb7hj\") pod \"payload-processing-e2e-autores-37448c-68dc44cbc9-j2789\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") " pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:17.018557 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.018534    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-plugins-config-volume\") pod \"payload-processing-e2e-autores-37448c-68dc44cbc9-j2789\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") " pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:17.026435 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.026405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7hj\" (UniqueName: \"kubernetes.io/projected/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-kube-api-access-tb7hj\") pod \"payload-processing-e2e-autores-37448c-68dc44cbc9-j2789\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") " pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:17.058453 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.058399    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:17.089612 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.089581    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:17.331374 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.331348    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"]
Sep 09 17:34:17.333729 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:34:17.333701    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60671309_799f_447d_a7ce_f4cf767c6547.slice/crio-a02b22a08f7ddd30b54da0f3e25ad8b60842f33937d77017270732386c1802c0 WatchSource:0}: Error finding container a02b22a08f7ddd30b54da0f3e25ad8b60842f33937d77017270732386c1802c0: Status 404 returned error can't find the container with id a02b22a08f7ddd30b54da0f3e25ad8b60842f33937d77017270732386c1802c0
Sep 09 17:34:17.375023 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:17.374997    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"]
Sep 09 17:34:17.376499 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:34:17.376461    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f845cc_3c15_4f8e_abd0_fa20f7c19f08.slice/crio-7489e74cf394a7df3b0ff781233c06c17b191b1a48c6d9bf600ea8d4cf241b63 WatchSource:0}: Error finding container 7489e74cf394a7df3b0ff781233c06c17b191b1a48c6d9bf600ea8d4cf241b63: Status 404 returned error can't find the container with id 7489e74cf394a7df3b0ff781233c06c17b191b1a48c6d9bf600ea8d4cf241b63
Sep 09 17:34:18.133657 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.133620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789" event={"ID":"78f845cc-3c15-4f8e-abd0-fa20f7c19f08","Type":"ContainerStarted","Data":"3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494"}
Sep 09 17:34:18.134184 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.133697    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789" event={"ID":"78f845cc-3c15-4f8e-abd0-fa20f7c19f08","Type":"ContainerStarted","Data":"7489e74cf394a7df3b0ff781233c06c17b191b1a48c6d9bf600ea8d4cf241b63"}
Sep 09 17:34:18.134184 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.133750    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:18.135567 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.135538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl" event={"ID":"60671309-799f-447d-a7ce-f4cf767c6547","Type":"ContainerStarted","Data":"68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7"}
Sep 09 17:34:18.135701 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.135584    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl" event={"ID":"60671309-799f-447d-a7ce-f4cf767c6547","Type":"ContainerStarted","Data":"a02b22a08f7ddd30b54da0f3e25ad8b60842f33937d77017270732386c1802c0"}
Sep 09 17:34:18.135701 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.135622    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:18.152454 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.152404    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789" podStartSLOduration=2.152385849 podStartE2EDuration="2.152385849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:18.151629256 +0000 UTC m=+1033.779781961" watchObservedRunningTime="2026-09-09 17:34:18.152385849 +0000 UTC m=+1033.780538548"
Sep 09 17:34:18.166377 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:18.166337    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl" podStartSLOduration=2.16632326 podStartE2EDuration="2.16632326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:18.165086766 +0000 UTC m=+1033.793239480" watchObservedRunningTime="2026-09-09 17:34:18.16632326 +0000 UTC m=+1033.794475962"
Sep 09 17:34:19.083168 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:19.083135    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:34:26.119797 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:26.119766    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:29.142923 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:29.142890    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:34:29.143353 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:29.142992    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:34:36.019481 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.019434    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"]
Sep 09 17:34:36.023199 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.023178    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.026498 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.026470    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-53da12\""
Sep 09 17:34:36.026609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.026508    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-53da12-dockercfg-bxwsm\""
Sep 09 17:34:36.034983 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.034935    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"]
Sep 09 17:34:36.191265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.191224    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.191265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.191267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqh5\" (UniqueName: \"kubernetes.io/projected/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-kube-api-access-hwqh5\") pod \"payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.292760 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.292664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.292760 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.292703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqh5\" (UniqueName: \"kubernetes.io/projected/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-kube-api-access-hwqh5\") pod \"payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.293490 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.293464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.303265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.303238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqh5\" (UniqueName: \"kubernetes.io/projected/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-kube-api-access-hwqh5\") pod \"payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.334867 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.334838    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:36.604534 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:36.604505    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"]
Sep 09 17:34:36.606648 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:34:36.606622    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc892746e_47d6_4c03_b0e1_47e2b1f8d0e7.slice/crio-ed172ab8fcef359692ab7a42daf4debcf18e4be0dd441c769994fb1268f4fa43 WatchSource:0}: Error finding container ed172ab8fcef359692ab7a42daf4debcf18e4be0dd441c769994fb1268f4fa43: Status 404 returned error can't find the container with id ed172ab8fcef359692ab7a42daf4debcf18e4be0dd441c769994fb1268f4fa43
Sep 09 17:34:37.239589 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:37.239548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw" event={"ID":"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7","Type":"ContainerStarted","Data":"002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e"}
Sep 09 17:34:37.239989 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:37.239635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw" event={"ID":"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7","Type":"ContainerStarted","Data":"ed172ab8fcef359692ab7a42daf4debcf18e4be0dd441c769994fb1268f4fa43"}
Sep 09 17:34:37.239989 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:37.239674    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:37.257470 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:37.257410    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw" podStartSLOduration=1.257390843 podStartE2EDuration="1.257390843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:37.254732987 +0000 UTC m=+1052.882885702" watchObservedRunningTime="2026-09-09 17:34:37.257390843 +0000 UTC m=+1052.885543546"
Sep 09 17:34:48.246946 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:48.246906    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:34:54.408408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.408376    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"]
Sep 09 17:34:54.408858 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.408639    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq" podUID="7743bc10-cb19-47c9-b07b-625f5373e4e3" containerName="payload-pre-processing" containerID="cri-o://d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec" gracePeriod=30
Sep 09 17:34:54.661661 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.661601    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:54.735038 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.734988    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmzjq\" (UniqueName: \"kubernetes.io/projected/7743bc10-cb19-47c9-b07b-625f5373e4e3-kube-api-access-cmzjq\") pod \"7743bc10-cb19-47c9-b07b-625f5373e4e3\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") "
Sep 09 17:34:54.735231 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.735150    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7743bc10-cb19-47c9-b07b-625f5373e4e3-plugins-config-volume\") pod \"7743bc10-cb19-47c9-b07b-625f5373e4e3\" (UID: \"7743bc10-cb19-47c9-b07b-625f5373e4e3\") "
Sep 09 17:34:54.735499 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.735469    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7743bc10-cb19-47c9-b07b-625f5373e4e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7743bc10-cb19-47c9-b07b-625f5373e4e3" (UID: "7743bc10-cb19-47c9-b07b-625f5373e4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:34:54.737347 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.737320    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7743bc10-cb19-47c9-b07b-625f5373e4e3-kube-api-access-cmzjq" (OuterVolumeSpecName: "kube-api-access-cmzjq") pod "7743bc10-cb19-47c9-b07b-625f5373e4e3" (UID: "7743bc10-cb19-47c9-b07b-625f5373e4e3"). InnerVolumeSpecName "kube-api-access-cmzjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:34:54.836196 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.836169    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7743bc10-cb19-47c9-b07b-625f5373e4e3-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:34:54.836196 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:54.836195    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzjq\" (UniqueName: \"kubernetes.io/projected/7743bc10-cb19-47c9-b07b-625f5373e4e3-kube-api-access-cmzjq\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:34:55.335658 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.335619    2558 generic.go:356] "Generic (PLEG): container finished" podID="7743bc10-cb19-47c9-b07b-625f5373e4e3" containerID="d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec" exitCode=0
Sep 09 17:34:55.335863 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.335683    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"
Sep 09 17:34:55.335863 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.335703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq" event={"ID":"7743bc10-cb19-47c9-b07b-625f5373e4e3","Type":"ContainerDied","Data":"d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec"}
Sep 09 17:34:55.335863 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.335755    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq" event={"ID":"7743bc10-cb19-47c9-b07b-625f5373e4e3","Type":"ContainerDied","Data":"83a361aced16cdacb301fdad3ea89e0dd805ff492245080246dbe206257e2fe4"}
Sep 09 17:34:55.335863 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.335778    2558 scope.go:117] "RemoveContainer" containerID="d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec"
Sep 09 17:34:55.344717 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.344699    2558 scope.go:117] "RemoveContainer" containerID="d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec"
Sep 09 17:34:55.345032 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:34:55.345011    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec\": container with ID starting with d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec not found: ID does not exist" containerID="d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec"
Sep 09 17:34:55.345083 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.345042    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec"} err="failed to get container status \"d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec\": rpc error: code = NotFound desc = could not find container \"d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec\": container with ID starting with d84cfa506e78aeeed3ca454dd9603d5d28bad187cc6a1a273d3277713001ffec not found: ID does not exist"
Sep 09 17:34:55.353783 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.353749    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"]
Sep 09 17:34:55.355872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:55.355852    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399f043b-84d58c467d-s7scq"]
Sep 09 17:34:56.816702 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:56.816672    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7743bc10-cb19-47c9-b07b-625f5373e4e3" path="/var/lib/kubelet/pods/7743bc10-cb19-47c9-b07b-625f5373e4e3/volumes"
Sep 09 17:34:58.847984 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.847918    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"]
Sep 09 17:34:58.848466 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.848442    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7743bc10-cb19-47c9-b07b-625f5373e4e3" containerName="payload-pre-processing"
Sep 09 17:34:58.848466 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.848460    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7743bc10-cb19-47c9-b07b-625f5373e4e3" containerName="payload-pre-processing"
Sep 09 17:34:58.848578 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.848537    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7743bc10-cb19-47c9-b07b-625f5373e4e3" containerName="payload-pre-processing"
Sep 09 17:34:58.852928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.852906    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:58.855721 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.855700    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1277f005-dockercfg-48szr\""
Sep 09 17:34:58.855833 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.855738    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1277f005\""
Sep 09 17:34:58.861936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.861907    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"]
Sep 09 17:34:58.871244 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.871219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2a5b3-ee65-4196-9071-d7474ed8f280-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:58.871383 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.871329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66n74\" (UniqueName: \"kubernetes.io/projected/4ca2a5b3-ee65-4196-9071-d7474ed8f280-kube-api-access-66n74\") pod \"payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:58.887843 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.887814    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"]
Sep 09 17:34:58.891285 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.891261    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:58.900796 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.900772    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"]
Sep 09 17:34:58.971944 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.971912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66n74\" (UniqueName: \"kubernetes.io/projected/4ca2a5b3-ee65-4196-9071-d7474ed8f280-kube-api-access-66n74\") pod \"payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:58.972136 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.971992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb341759-e389-4111-8cb8-dfbda15b9c8b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") " pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:58.972136 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.972017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2a5b3-ee65-4196-9071-d7474ed8f280-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:58.972136 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.972036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgxn\" (UniqueName: \"kubernetes.io/projected/bb341759-e389-4111-8cb8-dfbda15b9c8b-kube-api-access-blgxn\") pod \"payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") " pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:58.972664 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.972645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2a5b3-ee65-4196-9071-d7474ed8f280-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:58.979995 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:58.979969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66n74\" (UniqueName: \"kubernetes.io/projected/4ca2a5b3-ee65-4196-9071-d7474ed8f280-kube-api-access-66n74\") pod \"payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:59.073511 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.073473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb341759-e389-4111-8cb8-dfbda15b9c8b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") " pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:59.073511 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.073519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgxn\" (UniqueName: \"kubernetes.io/projected/bb341759-e389-4111-8cb8-dfbda15b9c8b-kube-api-access-blgxn\") pod \"payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") " pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:59.074239 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.074207    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb341759-e389-4111-8cb8-dfbda15b9c8b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") " pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:59.082881 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.082856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgxn\" (UniqueName: \"kubernetes.io/projected/bb341759-e389-4111-8cb8-dfbda15b9c8b-kube-api-access-blgxn\") pod \"payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") " pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:59.163866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.163787    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:34:59.202872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.202835    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:34:59.439685 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.439543    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"]
Sep 09 17:34:59.486652 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:34:59.486629    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"]
Sep 09 17:34:59.491697 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:34:59.491668    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb341759_e389_4111_8cb8_dfbda15b9c8b.slice/crio-adfcf9270296655956be1f2f97400f8134eb1d1ea4d681667d32d0451070ab71 WatchSource:0}: Error finding container adfcf9270296655956be1f2f97400f8134eb1d1ea4d681667d32d0451070ab71: Status 404 returned error can't find the container with id adfcf9270296655956be1f2f97400f8134eb1d1ea4d681667d32d0451070ab71
Sep 09 17:35:00.364172 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.364138    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h" event={"ID":"4ca2a5b3-ee65-4196-9071-d7474ed8f280","Type":"ContainerStarted","Data":"62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c"}
Sep 09 17:35:00.364661 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.364342    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:35:00.364661 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.364388    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h" event={"ID":"4ca2a5b3-ee65-4196-9071-d7474ed8f280","Type":"ContainerStarted","Data":"a3c84e53e522bc7f93d4a7ea5f1b339c7ee5fbab36e5877912f139d0c2daa8e1"}
Sep 09 17:35:00.365931 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.365877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm" event={"ID":"bb341759-e389-4111-8cb8-dfbda15b9c8b","Type":"ContainerStarted","Data":"07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0"}
Sep 09 17:35:00.365931 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.365924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm" event={"ID":"bb341759-e389-4111-8cb8-dfbda15b9c8b","Type":"ContainerStarted","Data":"adfcf9270296655956be1f2f97400f8134eb1d1ea4d681667d32d0451070ab71"}
Sep 09 17:35:00.366143 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.365999    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:35:00.380668 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.380613    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h" podStartSLOduration=2.380598564 podStartE2EDuration="2.380598564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:00.379403906 +0000 UTC m=+1076.007556610" watchObservedRunningTime="2026-09-09 17:35:00.380598564 +0000 UTC m=+1076.008751266"
Sep 09 17:35:00.394331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.394283    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm" podStartSLOduration=2.394267539 podStartE2EDuration="2.394267539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:00.394046062 +0000 UTC m=+1076.022198766" watchObservedRunningTime="2026-09-09 17:35:00.394267539 +0000 UTC m=+1076.022420243"
Sep 09 17:35:00.879433 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.879395    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"]
Sep 09 17:35:00.883127 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.883101    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:00.886313 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.886290    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-959540a1-kserve-self-signed-certs\""
Sep 09 17:35:00.894594 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.894570    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"]
Sep 09 17:35:00.991725 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.991690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-home\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:00.991725 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.991725    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-dshm\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:00.991925 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.991751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96znl\" (UniqueName: \"kubernetes.io/projected/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-kube-api-access-96znl\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:00.991925 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.991814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-model-cache\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:00.991925 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.991847    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tls-certs\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:00.991925 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:00.991884    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tmp-dir\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093393 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-home\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093569 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-dshm\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093569 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96znl\" (UniqueName: \"kubernetes.io/projected/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-kube-api-access-96znl\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093569 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-model-cache\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093569 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093501    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tls-certs\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093569 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tmp-dir\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093831 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-home\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.093935 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-model-cache\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.094030 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.093997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tmp-dir\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.095706 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.095685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-dshm\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.095844 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.095825    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tls-certs\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.102310 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.102289    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96znl\" (UniqueName: \"kubernetes.io/projected/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-kube-api-access-96znl\") pod \"e2e-conflict-a-959540a1-kserve-579454f675-jt99c\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") " pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.193703 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.193663    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:01.468207 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:01.468182    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"]
Sep 09 17:35:01.469713 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:35:01.469684    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ae472cc_4a6b_4d97_ba42_e745f19e0f06.slice/crio-afe6b2b1a6d78aa7856cb9111747732f3459ce710046494187578bcb84912a9f WatchSource:0}: Error finding container afe6b2b1a6d78aa7856cb9111747732f3459ce710046494187578bcb84912a9f: Status 404 returned error can't find the container with id afe6b2b1a6d78aa7856cb9111747732f3459ce710046494187578bcb84912a9f
Sep 09 17:35:02.378985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:02.378932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c" event={"ID":"2ae472cc-4a6b-4d97-ba42-e745f19e0f06","Type":"ContainerStarted","Data":"5558364c717d6ec9fd53b80baa6cea77ca60306ee6f4af4b93aebb84f9b73f83"}
Sep 09 17:35:02.378985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:02.378987    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c" event={"ID":"2ae472cc-4a6b-4d97-ba42-e745f19e0f06","Type":"ContainerStarted","Data":"afe6b2b1a6d78aa7856cb9111747732f3459ce710046494187578bcb84912a9f"}
Sep 09 17:35:02.395066 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:02.395008    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c" podStartSLOduration=2.394992626 podStartE2EDuration="2.394992626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:02.394017188 +0000 UTC m=+1078.022169891" watchObservedRunningTime="2026-09-09 17:35:02.394992626 +0000 UTC m=+1078.023145334"
Sep 09 17:35:11.194322 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:11.194270    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:11.194711 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:11.194342    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:11.206883 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:11.206857    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:11.373313 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:11.373284    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:35:11.373551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:11.373427    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:35:11.437592 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:11.437564    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:35:22.077348 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.077313    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"]
Sep 09 17:35:22.080990 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.080969    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.083718 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.083690    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-37448c\"/\"kube-root-ca.crt\""
Sep 09 17:35:22.083819 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.083730    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-37448c\"/\"openshift-service-ca.crt\""
Sep 09 17:35:22.085031 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.085010    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-37448c\"/\"default-dockercfg-hkmhk\""
Sep 09 17:35:22.085184 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.085164    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-37448c\"/\"autores-model-37448c-kserve-self-signed-certs\""
Sep 09 17:35:22.092430 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.092403    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"]
Sep 09 17:35:22.189540 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.189501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-model-cache\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.189700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.189597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tmp-dir\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.189700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.189628    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tls-certs\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.189700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.189655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-dshm\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.189832 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.189762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbknf\" (UniqueName: \"kubernetes.io/projected/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-kube-api-access-xbknf\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.189832 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.189792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-home\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.290695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.290659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-model-cache\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.290872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.290711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tmp-dir\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.290872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.290742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tls-certs\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.290872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.290760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-dshm\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.290872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.290797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbknf\" (UniqueName: \"kubernetes.io/projected/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-kube-api-access-xbknf\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.290872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.290835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-home\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.291192 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.291166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-model-cache\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.291248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.291226    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tmp-dir\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.291314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.291302    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-home\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.293215 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.293185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-dshm\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.293463 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.293442    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tls-certs\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.300968 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.300935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbknf\" (UniqueName: \"kubernetes.io/projected/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-kube-api-access-xbknf\") pod \"autores-model-37448c-kserve-558fc4b6c9-qzj9l\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") " pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.391808 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.391719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:22.680692 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.680665    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"]
Sep 09 17:35:22.681660 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:35:22.681625    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e9e3292_f98a_4128_b3b0_1d8cdb9833eb.slice/crio-6eb26f18114632793ca34be8525f18c2f12ab1dc2732a0c39974a0a902dbb3bc WatchSource:0}: Error finding container 6eb26f18114632793ca34be8525f18c2f12ab1dc2732a0c39974a0a902dbb3bc: Status 404 returned error can't find the container with id 6eb26f18114632793ca34be8525f18c2f12ab1dc2732a0c39974a0a902dbb3bc
Sep 09 17:35:22.751548 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.751519    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"]
Sep 09 17:35:22.755219 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.755203    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"
Sep 09 17:35:22.757793 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.757776    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:35:22.764435 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.764412    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"]
Sep 09 17:35:22.897317 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.897267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615e533b-0281-4879-a850-dbfe1476ecd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq\" (UID: \"615e533b-0281-4879-a850-dbfe1476ecd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"
Sep 09 17:35:22.998731 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.998652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615e533b-0281-4879-a850-dbfe1476ecd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq\" (UID: \"615e533b-0281-4879-a850-dbfe1476ecd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"
Sep 09 17:35:22.999342 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:22.999319    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615e533b-0281-4879-a850-dbfe1476ecd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq\" (UID: \"615e533b-0281-4879-a850-dbfe1476ecd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"
Sep 09 17:35:23.067464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:23.067427    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"
Sep 09 17:35:23.339474 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:23.339445    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"]
Sep 09 17:35:23.342247 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:35:23.342214    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615e533b_0281_4879_a850_dbfe1476ecd9.slice/crio-ba5a3b2a0440a7c75ca883fc6f1bd3eb9250b6e84b8ce90de5121bd2d38d11a2 WatchSource:0}: Error finding container ba5a3b2a0440a7c75ca883fc6f1bd3eb9250b6e84b8ce90de5121bd2d38d11a2: Status 404 returned error can't find the container with id ba5a3b2a0440a7c75ca883fc6f1bd3eb9250b6e84b8ce90de5121bd2d38d11a2
Sep 09 17:35:23.488678 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:23.488634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l" event={"ID":"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb","Type":"ContainerStarted","Data":"3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e"}
Sep 09 17:35:23.488870 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:23.488684    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l" event={"ID":"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb","Type":"ContainerStarted","Data":"6eb26f18114632793ca34be8525f18c2f12ab1dc2732a0c39974a0a902dbb3bc"}
Sep 09 17:35:23.489810 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:23.489784    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq" event={"ID":"615e533b-0281-4879-a850-dbfe1476ecd9","Type":"ContainerStarted","Data":"ba5a3b2a0440a7c75ca883fc6f1bd3eb9250b6e84b8ce90de5121bd2d38d11a2"}
Sep 09 17:35:23.508686 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:23.508631    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l" podStartSLOduration=1.5086128730000001 podStartE2EDuration="1.508612873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:23.507098006 +0000 UTC m=+1099.135250713" watchObservedRunningTime="2026-09-09 17:35:23.508612873 +0000 UTC m=+1099.136765574"
Sep 09 17:35:26.508438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:26.508347    2558 generic.go:356] "Generic (PLEG): container finished" podID="615e533b-0281-4879-a850-dbfe1476ecd9" containerID="ce0e76d4779ca7f926503aa8368e2c1b871ae1ae61ebbec0351dabffbc006ef4" exitCode=0
Sep 09 17:35:26.508790 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:26.508437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq" event={"ID":"615e533b-0281-4879-a850-dbfe1476ecd9","Type":"ContainerDied","Data":"ce0e76d4779ca7f926503aa8368e2c1b871ae1ae61ebbec0351dabffbc006ef4"}
Sep 09 17:35:27.650934 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:27.650905    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"
Sep 09 17:35:27.738738 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:27.738702    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615e533b-0281-4879-a850-dbfe1476ecd9-maas-api-service-ca\") pod \"615e533b-0281-4879-a850-dbfe1476ecd9\" (UID: \"615e533b-0281-4879-a850-dbfe1476ecd9\") "
Sep 09 17:35:27.739167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:27.739139    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615e533b-0281-4879-a850-dbfe1476ecd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "615e533b-0281-4879-a850-dbfe1476ecd9" (UID: "615e533b-0281-4879-a850-dbfe1476ecd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:35:27.839728 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:27.839656    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615e533b-0281-4879-a850-dbfe1476ecd9-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:35:28.522363 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:28.522325    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"
Sep 09 17:35:28.522546 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:28.522337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq" event={"ID":"615e533b-0281-4879-a850-dbfe1476ecd9","Type":"ContainerDied","Data":"ba5a3b2a0440a7c75ca883fc6f1bd3eb9250b6e84b8ce90de5121bd2d38d11a2"}
Sep 09 17:35:28.522546 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:28.522486    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba5a3b2a0440a7c75ca883fc6f1bd3eb9250b6e84b8ce90de5121bd2d38d11a2"
Sep 09 17:35:29.184412 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.184373    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"]
Sep 09 17:35:29.184922 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.184724    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615e533b-0281-4879-a850-dbfe1476ecd9" containerName="revoke-keys"
Sep 09 17:35:29.184922 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.184738    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="615e533b-0281-4879-a850-dbfe1476ecd9" containerName="revoke-keys"
Sep 09 17:35:29.184922 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.184818    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="615e533b-0281-4879-a850-dbfe1476ecd9" containerName="revoke-keys"
Sep 09 17:35:29.188145 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.188121    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.190756 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.190732    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ea996c\"/\"kube-root-ca.crt\""
Sep 09 17:35:29.191815 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.191791    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ea996c\"/\"default-dockercfg-4hjgm\""
Sep 09 17:35:29.192024 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.191893    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ea996c\"/\"openshift-service-ca.crt\""
Sep 09 17:35:29.192290 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.192273    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ea996c\"/\"test-model-ea996c-kserve-self-signed-certs\""
Sep 09 17:35:29.194988 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.194965    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"]
Sep 09 17:35:29.354382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.354343    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-home\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.354559 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.354388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-tmp-dir\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.354559 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.354415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31430376-c599-4c28-bc72-ffcb86b7da9c-tls-certs\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.354559 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.354493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-model-cache\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.354559 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.354520    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-dshm\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.354559 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.354538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ggrp\" (UniqueName: \"kubernetes.io/projected/31430376-c599-4c28-bc72-ffcb86b7da9c-kube-api-access-2ggrp\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.455571 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.455463    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-model-cache\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.455571 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.455556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-dshm\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.455797 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.455589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ggrp\" (UniqueName: \"kubernetes.io/projected/31430376-c599-4c28-bc72-ffcb86b7da9c-kube-api-access-2ggrp\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.455884 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.455720    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-home\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.456000 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.455971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-tmp-dir\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.456134 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.456035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31430376-c599-4c28-bc72-ffcb86b7da9c-tls-certs\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.456376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.456231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-model-cache\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.456376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.456310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-home\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.456517 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.456482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-tmp-dir\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.458116 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.458095    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-dshm\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.458439 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.458422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31430376-c599-4c28-bc72-ffcb86b7da9c-tls-certs\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.463063 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.463034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ggrp\" (UniqueName: \"kubernetes.io/projected/31430376-c599-4c28-bc72-ffcb86b7da9c-kube-api-access-2ggrp\") pod \"test-model-ea996c-kserve-5bf5ff567-q52hd\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") " pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.499185 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.499154    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:29.530806 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.530770    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"]
Sep 09 17:35:29.531312 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.531045    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm" podUID="bb341759-e389-4111-8cb8-dfbda15b9c8b" containerName="payload-processing" containerID="cri-o://07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0" gracePeriod=30
Sep 09 17:35:29.540771 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.540641    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"]
Sep 09 17:35:29.541114 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.540917    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h" podUID="4ca2a5b3-ee65-4196-9071-d7474ed8f280" containerName="payload-pre-processing" containerID="cri-o://62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c" gracePeriod=30
Sep 09 17:35:29.696104 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.696041    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"]
Sep 09 17:35:29.699971 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.699933    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1277f005-zj6xq"]
Sep 09 17:35:29.791976 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.791853    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"]
Sep 09 17:35:29.793144 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:35:29.793116    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31430376_c599_4c28_bc72_ffcb86b7da9c.slice/crio-385b9038a5e917f06d9e064220959913ab2ab147293c04e6d4d1590fef7bcaa3 WatchSource:0}: Error finding container 385b9038a5e917f06d9e064220959913ab2ab147293c04e6d4d1590fef7bcaa3: Status 404 returned error can't find the container with id 385b9038a5e917f06d9e064220959913ab2ab147293c04e6d4d1590fef7bcaa3
Sep 09 17:35:29.800253 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.800236    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:35:29.841518 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.841490    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:35:29.959831 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.959801    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66n74\" (UniqueName: \"kubernetes.io/projected/4ca2a5b3-ee65-4196-9071-d7474ed8f280-kube-api-access-66n74\") pod \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") "
Sep 09 17:35:29.960025 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.959840    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blgxn\" (UniqueName: \"kubernetes.io/projected/bb341759-e389-4111-8cb8-dfbda15b9c8b-kube-api-access-blgxn\") pod \"bb341759-e389-4111-8cb8-dfbda15b9c8b\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") "
Sep 09 17:35:29.960025 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.959888    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb341759-e389-4111-8cb8-dfbda15b9c8b-plugins-config-volume\") pod \"bb341759-e389-4111-8cb8-dfbda15b9c8b\" (UID: \"bb341759-e389-4111-8cb8-dfbda15b9c8b\") "
Sep 09 17:35:29.960025 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.959913    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2a5b3-ee65-4196-9071-d7474ed8f280-plugins-config-volume\") pod \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\" (UID: \"4ca2a5b3-ee65-4196-9071-d7474ed8f280\") "
Sep 09 17:35:29.960331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.960294    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb341759-e389-4111-8cb8-dfbda15b9c8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb341759-e389-4111-8cb8-dfbda15b9c8b" (UID: "bb341759-e389-4111-8cb8-dfbda15b9c8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:35:29.960434 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.960324    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca2a5b3-ee65-4196-9071-d7474ed8f280-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ca2a5b3-ee65-4196-9071-d7474ed8f280" (UID: "4ca2a5b3-ee65-4196-9071-d7474ed8f280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:35:29.961883 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.961862    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb341759-e389-4111-8cb8-dfbda15b9c8b-kube-api-access-blgxn" (OuterVolumeSpecName: "kube-api-access-blgxn") pod "bb341759-e389-4111-8cb8-dfbda15b9c8b" (UID: "bb341759-e389-4111-8cb8-dfbda15b9c8b"). InnerVolumeSpecName "kube-api-access-blgxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:35:29.961949 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:29.961878    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca2a5b3-ee65-4196-9071-d7474ed8f280-kube-api-access-66n74" (OuterVolumeSpecName: "kube-api-access-66n74") pod "4ca2a5b3-ee65-4196-9071-d7474ed8f280" (UID: "4ca2a5b3-ee65-4196-9071-d7474ed8f280"). InnerVolumeSpecName "kube-api-access-66n74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:35:30.061493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.061410    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb341759-e389-4111-8cb8-dfbda15b9c8b-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:35:30.061493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.061438    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca2a5b3-ee65-4196-9071-d7474ed8f280-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:35:30.061493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.061447    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66n74\" (UniqueName: \"kubernetes.io/projected/4ca2a5b3-ee65-4196-9071-d7474ed8f280-kube-api-access-66n74\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:35:30.061493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.061457    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgxn\" (UniqueName: \"kubernetes.io/projected/bb341759-e389-4111-8cb8-dfbda15b9c8b-kube-api-access-blgxn\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:35:30.537687 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.537647    2558 generic.go:356] "Generic (PLEG): container finished" podID="4ca2a5b3-ee65-4196-9071-d7474ed8f280" containerID="62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c" exitCode=0
Sep 09 17:35:30.538173 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.537703    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"
Sep 09 17:35:30.538173 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.537733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h" event={"ID":"4ca2a5b3-ee65-4196-9071-d7474ed8f280","Type":"ContainerDied","Data":"62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c"}
Sep 09 17:35:30.538173 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.537795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h" event={"ID":"4ca2a5b3-ee65-4196-9071-d7474ed8f280","Type":"ContainerDied","Data":"a3c84e53e522bc7f93d4a7ea5f1b339c7ee5fbab36e5877912f139d0c2daa8e1"}
Sep 09 17:35:30.538173 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.537825    2558 scope.go:117] "RemoveContainer" containerID="62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c"
Sep 09 17:35:30.539468 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.539417    2558 generic.go:356] "Generic (PLEG): container finished" podID="bb341759-e389-4111-8cb8-dfbda15b9c8b" containerID="07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0" exitCode=0
Sep 09 17:35:30.539564 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.539484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm" event={"ID":"bb341759-e389-4111-8cb8-dfbda15b9c8b","Type":"ContainerDied","Data":"07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0"}
Sep 09 17:35:30.539564 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.539516    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"
Sep 09 17:35:30.539564 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.539548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm" event={"ID":"bb341759-e389-4111-8cb8-dfbda15b9c8b","Type":"ContainerDied","Data":"adfcf9270296655956be1f2f97400f8134eb1d1ea4d681667d32d0451070ab71"}
Sep 09 17:35:30.541971 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.541935    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd" event={"ID":"31430376-c599-4c28-bc72-ffcb86b7da9c","Type":"ContainerStarted","Data":"fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc"}
Sep 09 17:35:30.542097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.542001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd" event={"ID":"31430376-c599-4c28-bc72-ffcb86b7da9c","Type":"ContainerStarted","Data":"385b9038a5e917f06d9e064220959913ab2ab147293c04e6d4d1590fef7bcaa3"}
Sep 09 17:35:30.550496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.550473    2558 scope.go:117] "RemoveContainer" containerID="62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c"
Sep 09 17:35:30.550834 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:35:30.550805    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c\": container with ID starting with 62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c not found: ID does not exist" containerID="62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c"
Sep 09 17:35:30.550926 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.550843    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c"} err="failed to get container status \"62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c\": rpc error: code = NotFound desc = could not find container \"62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c\": container with ID starting with 62fef464ffdc132a3f99c5ec15984072f8d98cf480aab2b4a245efa54b862f1c not found: ID does not exist"
Sep 09 17:35:30.550926 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.550864    2558 scope.go:117] "RemoveContainer" containerID="07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0"
Sep 09 17:35:30.561081 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.561059    2558 scope.go:117] "RemoveContainer" containerID="07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0"
Sep 09 17:35:30.561420 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:35:30.561380    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0\": container with ID starting with 07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0 not found: ID does not exist" containerID="07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0"
Sep 09 17:35:30.561487 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.561427    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0"} err="failed to get container status \"07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0\": rpc error: code = NotFound desc = could not find container \"07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0\": container with ID starting with 07756771f58ee42f8e1166a95ea1c2e6859abfaadf839c66f454dddf40ca5cd0 not found: ID does not exist"
Sep 09 17:35:30.572171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.572081    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd" podStartSLOduration=1.5720596009999999 podStartE2EDuration="1.572059601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:30.567302618 +0000 UTC m=+1106.195455324" watchObservedRunningTime="2026-09-09 17:35:30.572059601 +0000 UTC m=+1106.200212306"
Sep 09 17:35:30.588828 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.588794    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"]
Sep 09 17:35:30.598592 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.598559    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1277f005-6dc8b6dc9d-2fl4h"]
Sep 09 17:35:30.608260 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.608228    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"]
Sep 09 17:35:30.611860 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.611831    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1277f005-cbb7fddf8-f5msm"]
Sep 09 17:35:30.816098 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.816020    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca2a5b3-ee65-4196-9071-d7474ed8f280" path="/var/lib/kubelet/pods/4ca2a5b3-ee65-4196-9071-d7474ed8f280/volumes"
Sep 09 17:35:30.816398 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.816385    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615e533b-0281-4879-a850-dbfe1476ecd9" path="/var/lib/kubelet/pods/615e533b-0281-4879-a850-dbfe1476ecd9/volumes"
Sep 09 17:35:30.816696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:30.816685    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb341759-e389-4111-8cb8-dfbda15b9c8b" path="/var/lib/kubelet/pods/bb341759-e389-4111-8cb8-dfbda15b9c8b/volumes"
Sep 09 17:35:32.392168 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:32.392128    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:32.392538 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:32.392191    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:32.405077 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:32.405044    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:32.566634 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:32.566603    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:35:34.651998 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.651963    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"]
Sep 09 17:35:34.652354 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.652303    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca2a5b3-ee65-4196-9071-d7474ed8f280" containerName="payload-pre-processing"
Sep 09 17:35:34.652354 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.652313    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca2a5b3-ee65-4196-9071-d7474ed8f280" containerName="payload-pre-processing"
Sep 09 17:35:34.652354 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.652323    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb341759-e389-4111-8cb8-dfbda15b9c8b" containerName="payload-processing"
Sep 09 17:35:34.652354 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.652329    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb341759-e389-4111-8cb8-dfbda15b9c8b" containerName="payload-processing"
Sep 09 17:35:34.652485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.652383    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb341759-e389-4111-8cb8-dfbda15b9c8b" containerName="payload-processing"
Sep 09 17:35:34.652485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.652389    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca2a5b3-ee65-4196-9071-d7474ed8f280" containerName="payload-pre-processing"
Sep 09 17:35:34.655579 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.655562    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:34.658808 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.658788    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-50082ca1-dockercfg-2zzvm\""
Sep 09 17:35:34.658928 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.658830    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-50082ca1\""
Sep 09 17:35:34.668966 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.668926    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"]
Sep 09 17:35:34.809158 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.809124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4glrt\" (UniqueName: \"kubernetes.io/projected/13668cd9-1538-4acb-8bca-7e69683a2ee4-kube-api-access-4glrt\") pod \"payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:34.809319 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.809174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13668cd9-1538-4acb-8bca-7e69683a2ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:34.909890 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.909796    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4glrt\" (UniqueName: \"kubernetes.io/projected/13668cd9-1538-4acb-8bca-7e69683a2ee4-kube-api-access-4glrt\") pod \"payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:34.909890 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.909846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13668cd9-1538-4acb-8bca-7e69683a2ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:34.910509 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.910489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13668cd9-1538-4acb-8bca-7e69683a2ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:34.917964 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.917925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4glrt\" (UniqueName: \"kubernetes.io/projected/13668cd9-1538-4acb-8bca-7e69683a2ee4-kube-api-access-4glrt\") pod \"payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:34.966134 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:34.966104    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:35.241819 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:35.241793    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"]
Sep 09 17:35:35.243307 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:35:35.243279    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13668cd9_1538_4acb_8bca_7e69683a2ee4.slice/crio-ad909b66fde9ce7f13daa7fe78bffbc29ab983e8d4aa1e57c2f719597b808f49 WatchSource:0}: Error finding container ad909b66fde9ce7f13daa7fe78bffbc29ab983e8d4aa1e57c2f719597b808f49: Status 404 returned error can't find the container with id ad909b66fde9ce7f13daa7fe78bffbc29ab983e8d4aa1e57c2f719597b808f49
Sep 09 17:35:35.576536 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:35.576431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l" event={"ID":"13668cd9-1538-4acb-8bca-7e69683a2ee4","Type":"ContainerStarted","Data":"32fbadd68c272cbd2e3f27623b59b5b6dc587acb504b4427d550b49e324e75e5"}
Sep 09 17:35:35.576536 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:35.576526    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:35.576728 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:35.576546    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l" event={"ID":"13668cd9-1538-4acb-8bca-7e69683a2ee4","Type":"ContainerStarted","Data":"ad909b66fde9ce7f13daa7fe78bffbc29ab983e8d4aa1e57c2f719597b808f49"}
Sep 09 17:35:35.593255 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:35.593191    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l" podStartSLOduration=1.593166853 podStartE2EDuration="1.593166853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:35.590659101 +0000 UTC m=+1111.218811827" watchObservedRunningTime="2026-09-09 17:35:35.593166853 +0000 UTC m=+1111.221319557"
Sep 09 17:35:39.499945 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:39.499902    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:39.500327 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:39.499986    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:39.512596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:39.512566    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:39.610335 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:39.610256    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:35:46.585127 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:46.585094    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:35:57.594523 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.594489    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"]
Sep 09 17:35:57.598075 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.598054    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"
Sep 09 17:35:57.600683 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.600657    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:35:57.605035 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.604990    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"]
Sep 09 17:35:57.715912 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.715876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-37448c-9dkh4\" (UID: \"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"
Sep 09 17:35:57.817025 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.816948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-37448c-9dkh4\" (UID: \"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"
Sep 09 17:35:57.817545 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.817522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-37448c-9dkh4\" (UID: \"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"
Sep 09 17:35:57.911616 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:57.911521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"
Sep 09 17:35:58.184647 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:58.184620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"]
Sep 09 17:35:58.186813 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:35:58.186774    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00abd50_01ce_485a_b3eb_3f1d8c2b1fc5.slice/crio-711dd08aa03214d5d7158e3aa7315908b6c501f07f38b3c57abadab67c315097 WatchSource:0}: Error finding container 711dd08aa03214d5d7158e3aa7315908b6c501f07f38b3c57abadab67c315097: Status 404 returned error can't find the container with id 711dd08aa03214d5d7158e3aa7315908b6c501f07f38b3c57abadab67c315097
Sep 09 17:35:58.712777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:58.712741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4" event={"ID":"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5","Type":"ContainerStarted","Data":"b7bc7ab238d3a15c492699bb46dd6a157983f330c3db97977a75d880654e3eb2"}
Sep 09 17:35:58.713187 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:58.712818    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4" event={"ID":"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5","Type":"ContainerStarted","Data":"711dd08aa03214d5d7158e3aa7315908b6c501f07f38b3c57abadab67c315097"}
Sep 09 17:35:58.729532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:58.729481    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4" podStartSLOduration=1.72946591 podStartE2EDuration="1.72946591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:58.727358687 +0000 UTC m=+1134.355511397" watchObservedRunningTime="2026-09-09 17:35:58.72946591 +0000 UTC m=+1134.357618613"
Sep 09 17:35:59.720260 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:59.720223    2558 generic.go:356] "Generic (PLEG): container finished" podID="b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5" containerID="b7bc7ab238d3a15c492699bb46dd6a157983f330c3db97977a75d880654e3eb2" exitCode=0
Sep 09 17:35:59.720675 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:35:59.720322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4" event={"ID":"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5","Type":"ContainerDied","Data":"b7bc7ab238d3a15c492699bb46dd6a157983f330c3db97977a75d880654e3eb2"}
Sep 09 17:36:00.863637 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:00.863607    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"
Sep 09 17:36:00.948608 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:00.948568    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5-maas-api-service-ca\") pod \"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5\" (UID: \"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5\") "
Sep 09 17:36:00.948943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:00.948919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5" (UID: "b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:36:01.049271 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:01.049181    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:01.733659 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:01.733627    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"
Sep 09 17:36:01.733659 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:01.733640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4" event={"ID":"b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5","Type":"ContainerDied","Data":"711dd08aa03214d5d7158e3aa7315908b6c501f07f38b3c57abadab67c315097"}
Sep 09 17:36:01.733918 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:01.733692    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711dd08aa03214d5d7158e3aa7315908b6c501f07f38b3c57abadab67c315097"
Sep 09 17:36:04.534419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.534382    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"]
Sep 09 17:36:04.534864 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.534622    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l" podUID="13668cd9-1538-4acb-8bca-7e69683a2ee4" containerName="payload-pre-processing" containerID="cri-o://32fbadd68c272cbd2e3f27623b59b5b6dc587acb504b4427d550b49e324e75e5" gracePeriod=30
Sep 09 17:36:04.751622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.751580    2558 generic.go:356] "Generic (PLEG): container finished" podID="13668cd9-1538-4acb-8bca-7e69683a2ee4" containerID="32fbadd68c272cbd2e3f27623b59b5b6dc587acb504b4427d550b49e324e75e5" exitCode=0
Sep 09 17:36:04.751814 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.751638    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l" event={"ID":"13668cd9-1538-4acb-8bca-7e69683a2ee4","Type":"ContainerDied","Data":"32fbadd68c272cbd2e3f27623b59b5b6dc587acb504b4427d550b49e324e75e5"}
Sep 09 17:36:04.787763 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.787709    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:36:04.888727 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.888683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4glrt\" (UniqueName: \"kubernetes.io/projected/13668cd9-1538-4acb-8bca-7e69683a2ee4-kube-api-access-4glrt\") pod \"13668cd9-1538-4acb-8bca-7e69683a2ee4\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") "
Sep 09 17:36:04.888927 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.888750    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13668cd9-1538-4acb-8bca-7e69683a2ee4-plugins-config-volume\") pod \"13668cd9-1538-4acb-8bca-7e69683a2ee4\" (UID: \"13668cd9-1538-4acb-8bca-7e69683a2ee4\") "
Sep 09 17:36:04.889174 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.889141    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13668cd9-1538-4acb-8bca-7e69683a2ee4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13668cd9-1538-4acb-8bca-7e69683a2ee4" (UID: "13668cd9-1538-4acb-8bca-7e69683a2ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:36:04.891053 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.891021    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13668cd9-1538-4acb-8bca-7e69683a2ee4-kube-api-access-4glrt" (OuterVolumeSpecName: "kube-api-access-4glrt") pod "13668cd9-1538-4acb-8bca-7e69683a2ee4" (UID: "13668cd9-1538-4acb-8bca-7e69683a2ee4"). InnerVolumeSpecName "kube-api-access-4glrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:36:04.990197 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.990156    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4glrt\" (UniqueName: \"kubernetes.io/projected/13668cd9-1538-4acb-8bca-7e69683a2ee4-kube-api-access-4glrt\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:04.990197 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:04.990188    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13668cd9-1538-4acb-8bca-7e69683a2ee4-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:05.758877 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:05.758843    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"
Sep 09 17:36:05.759392 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:05.758883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l" event={"ID":"13668cd9-1538-4acb-8bca-7e69683a2ee4","Type":"ContainerDied","Data":"ad909b66fde9ce7f13daa7fe78bffbc29ab983e8d4aa1e57c2f719597b808f49"}
Sep 09 17:36:05.759392 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:05.758946    2558 scope.go:117] "RemoveContainer" containerID="32fbadd68c272cbd2e3f27623b59b5b6dc587acb504b4427d550b49e324e75e5"
Sep 09 17:36:05.786552 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:05.786519    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"]
Sep 09 17:36:05.804314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:05.804285    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50082ca1-97b7c5fb6-5c84l"]
Sep 09 17:36:06.817210 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:06.817168    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13668cd9-1538-4acb-8bca-7e69683a2ee4" path="/var/lib/kubelet/pods/13668cd9-1538-4acb-8bca-7e69683a2ee4/volumes"
Sep 09 17:36:08.444672 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.444640    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"]
Sep 09 17:36:08.445127 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.444879    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789" podUID="78f845cc-3c15-4f8e-abd0-fa20f7c19f08" containerName="payload-processing" containerID="cri-o://3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494" gracePeriod=30
Sep 09 17:36:08.449385 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.449362    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"]
Sep 09 17:36:08.449743 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.449711    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl" podUID="60671309-799f-447d-a7ce-f4cf767c6547" containerName="payload-pre-processing" containerID="cri-o://68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7" gracePeriod=30
Sep 09 17:36:08.599065 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.599014    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"]
Sep 09 17:36:08.601271 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.601248    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-37448c-9dkh4"]
Sep 09 17:36:08.705766 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.705703    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:36:08.728837 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.728814    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:36:08.777624 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.777583    2558 generic.go:356] "Generic (PLEG): container finished" podID="78f845cc-3c15-4f8e-abd0-fa20f7c19f08" containerID="3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494" exitCode=0
Sep 09 17:36:08.777624 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.777622    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"
Sep 09 17:36:08.777861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.777659    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789" event={"ID":"78f845cc-3c15-4f8e-abd0-fa20f7c19f08","Type":"ContainerDied","Data":"3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494"}
Sep 09 17:36:08.777861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.777711    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789" event={"ID":"78f845cc-3c15-4f8e-abd0-fa20f7c19f08","Type":"ContainerDied","Data":"7489e74cf394a7df3b0ff781233c06c17b191b1a48c6d9bf600ea8d4cf241b63"}
Sep 09 17:36:08.777861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.777745    2558 scope.go:117] "RemoveContainer" containerID="3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494"
Sep 09 17:36:08.779070 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.779039    2558 generic.go:356] "Generic (PLEG): container finished" podID="60671309-799f-447d-a7ce-f4cf767c6547" containerID="68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7" exitCode=0
Sep 09 17:36:08.779188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.779101    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"
Sep 09 17:36:08.779188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.779126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl" event={"ID":"60671309-799f-447d-a7ce-f4cf767c6547","Type":"ContainerDied","Data":"68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7"}
Sep 09 17:36:08.779188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.779174    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl" event={"ID":"60671309-799f-447d-a7ce-f4cf767c6547","Type":"ContainerDied","Data":"a02b22a08f7ddd30b54da0f3e25ad8b60842f33937d77017270732386c1802c0"}
Sep 09 17:36:08.787359 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.787341    2558 scope.go:117] "RemoveContainer" containerID="3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494"
Sep 09 17:36:08.787634 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:36:08.787615    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494\": container with ID starting with 3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494 not found: ID does not exist" containerID="3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494"
Sep 09 17:36:08.787723 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.787649    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494"} err="failed to get container status \"3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494\": rpc error: code = NotFound desc = could not find container \"3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494\": container with ID starting with 3a72a69b48b2ba89fce5d296087346f0c1ae677e22efc16f6339a41ce2972494 not found: ID does not exist"
Sep 09 17:36:08.787723 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.787675    2558 scope.go:117] "RemoveContainer" containerID="68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7"
Sep 09 17:36:08.796435 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.796416    2558 scope.go:117] "RemoveContainer" containerID="68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7"
Sep 09 17:36:08.796723 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:36:08.796694    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7\": container with ID starting with 68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7 not found: ID does not exist" containerID="68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7"
Sep 09 17:36:08.796819 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.796722    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7"} err="failed to get container status \"68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7\": rpc error: code = NotFound desc = could not find container \"68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7\": container with ID starting with 68b917181584e9a715f31e4a4dbcfa45cffa8d10aad55c3c18162cc5f0b3f9e7 not found: ID does not exist"
Sep 09 17:36:08.818085 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.818054    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5" path="/var/lib/kubelet/pods/b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5/volumes"
Sep 09 17:36:08.824560 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.824534    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb7hj\" (UniqueName: \"kubernetes.io/projected/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-kube-api-access-tb7hj\") pod \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") "
Sep 09 17:36:08.824674 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.824577    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60671309-799f-447d-a7ce-f4cf767c6547-plugins-config-volume\") pod \"60671309-799f-447d-a7ce-f4cf767c6547\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") "
Sep 09 17:36:08.824674 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.824626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-plugins-config-volume\") pod \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\" (UID: \"78f845cc-3c15-4f8e-abd0-fa20f7c19f08\") "
Sep 09 17:36:08.824674 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.824647    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wtlv\" (UniqueName: \"kubernetes.io/projected/60671309-799f-447d-a7ce-f4cf767c6547-kube-api-access-9wtlv\") pod \"60671309-799f-447d-a7ce-f4cf767c6547\" (UID: \"60671309-799f-447d-a7ce-f4cf767c6547\") "
Sep 09 17:36:08.824995 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.824969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60671309-799f-447d-a7ce-f4cf767c6547-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60671309-799f-447d-a7ce-f4cf767c6547" (UID: "60671309-799f-447d-a7ce-f4cf767c6547"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:36:08.825059 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.824988    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78f845cc-3c15-4f8e-abd0-fa20f7c19f08" (UID: "78f845cc-3c15-4f8e-abd0-fa20f7c19f08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:36:08.826594 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.826574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-kube-api-access-tb7hj" (OuterVolumeSpecName: "kube-api-access-tb7hj") pod "78f845cc-3c15-4f8e-abd0-fa20f7c19f08" (UID: "78f845cc-3c15-4f8e-abd0-fa20f7c19f08"). InnerVolumeSpecName "kube-api-access-tb7hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:36:08.826642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.826627    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60671309-799f-447d-a7ce-f4cf767c6547-kube-api-access-9wtlv" (OuterVolumeSpecName: "kube-api-access-9wtlv") pod "60671309-799f-447d-a7ce-f4cf767c6547" (UID: "60671309-799f-447d-a7ce-f4cf767c6547"). InnerVolumeSpecName "kube-api-access-9wtlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:36:08.925917 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.925876    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb7hj\" (UniqueName: \"kubernetes.io/projected/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-kube-api-access-tb7hj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:08.925917 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.925906    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60671309-799f-447d-a7ce-f4cf767c6547-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:08.925917 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.925918    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f845cc-3c15-4f8e-abd0-fa20f7c19f08-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:08.925917 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:08.925929    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wtlv\" (UniqueName: \"kubernetes.io/projected/60671309-799f-447d-a7ce-f4cf767c6547-kube-api-access-9wtlv\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:09.102191 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:09.102151    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"]
Sep 09 17:36:09.105497 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:09.105471    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-37448c-68dc44cbc9-j2789"]
Sep 09 17:36:09.116499 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:09.116472    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"]
Sep 09 17:36:09.122920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:09.122894    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37448c-68f49d6566-96nhl"]
Sep 09 17:36:10.772641 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772603    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"]
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772916    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f845cc-3c15-4f8e-abd0-fa20f7c19f08" containerName="payload-processing"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772928    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f845cc-3c15-4f8e-abd0-fa20f7c19f08" containerName="payload-processing"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772949    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60671309-799f-447d-a7ce-f4cf767c6547" containerName="payload-pre-processing"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772968    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="60671309-799f-447d-a7ce-f4cf767c6547" containerName="payload-pre-processing"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772974    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13668cd9-1538-4acb-8bca-7e69683a2ee4" containerName="payload-pre-processing"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772980    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="13668cd9-1538-4acb-8bca-7e69683a2ee4" containerName="payload-pre-processing"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772987    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5" containerName="revoke-keys"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.772992    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5" containerName="revoke-keys"
Sep 09 17:36:10.773048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.773045    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f845cc-3c15-4f8e-abd0-fa20f7c19f08" containerName="payload-processing"
Sep 09 17:36:10.773330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.773054    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b00abd50-01ce-485a-b3eb-3f1d8c2b1fc5" containerName="revoke-keys"
Sep 09 17:36:10.773330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.773061    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="60671309-799f-447d-a7ce-f4cf767c6547" containerName="payload-pre-processing"
Sep 09 17:36:10.773330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.773068    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="13668cd9-1538-4acb-8bca-7e69683a2ee4" containerName="payload-pre-processing"
Sep 09 17:36:10.777374 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.777352    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:10.780124 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.780101    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0aa81a54-dockercfg-rs2fx\""
Sep 09 17:36:10.780217 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.780132    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0aa81a54\""
Sep 09 17:36:10.786300 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.786277    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"]
Sep 09 17:36:10.809400 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.809365    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"]
Sep 09 17:36:10.812690 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.812673    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:10.829498 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.829465    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60671309-799f-447d-a7ce-f4cf767c6547" path="/var/lib/kubelet/pods/60671309-799f-447d-a7ce-f4cf767c6547/volumes"
Sep 09 17:36:10.829839 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.829818    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f845cc-3c15-4f8e-abd0-fa20f7c19f08" path="/var/lib/kubelet/pods/78f845cc-3c15-4f8e-abd0-fa20f7c19f08/volumes"
Sep 09 17:36:10.830163 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.830123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"]
Sep 09 17:36:10.944102 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.944062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8wc\" (UniqueName: \"kubernetes.io/projected/7879e28f-0d6a-4713-a209-a40c46eabd56-kube-api-access-nl8wc\") pod \"payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:10.944102 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.944106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4024d5-a487-491a-86fd-9f2cf7bca895-plugins-config-volume\") pod \"payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") " pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:10.944331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.944161    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e28f-0d6a-4713-a209-a40c46eabd56-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:10.944331 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:10.944321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h666l\" (UniqueName: \"kubernetes.io/projected/cb4024d5-a487-491a-86fd-9f2cf7bca895-kube-api-access-h666l\") pod \"payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") " pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:11.045261 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.045173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h666l\" (UniqueName: \"kubernetes.io/projected/cb4024d5-a487-491a-86fd-9f2cf7bca895-kube-api-access-h666l\") pod \"payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") " pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:11.045261 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.045248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8wc\" (UniqueName: \"kubernetes.io/projected/7879e28f-0d6a-4713-a209-a40c46eabd56-kube-api-access-nl8wc\") pod \"payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:11.045488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.045272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4024d5-a487-491a-86fd-9f2cf7bca895-plugins-config-volume\") pod \"payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") " pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:11.045488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.045289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e28f-0d6a-4713-a209-a40c46eabd56-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:11.046031 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.046009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e28f-0d6a-4713-a209-a40c46eabd56-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:11.046131 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.046028    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4024d5-a487-491a-86fd-9f2cf7bca895-plugins-config-volume\") pod \"payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") " pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:11.052663 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.052632    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h666l\" (UniqueName: \"kubernetes.io/projected/cb4024d5-a487-491a-86fd-9f2cf7bca895-kube-api-access-h666l\") pod \"payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") " pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:11.052663 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.052648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8wc\" (UniqueName: \"kubernetes.io/projected/7879e28f-0d6a-4713-a209-a40c46eabd56-kube-api-access-nl8wc\") pod \"payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:11.087650 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.087619    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:11.130093 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.130044    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:11.366055 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.366025    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"]
Sep 09 17:36:11.366603 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:36:11.366582    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7879e28f_0d6a_4713_a209_a40c46eabd56.slice/crio-1798fc45effa7f5a804aea00316c3a610725b293c9df80f302013a0aaff9e4a7 WatchSource:0}: Error finding container 1798fc45effa7f5a804aea00316c3a610725b293c9df80f302013a0aaff9e4a7: Status 404 returned error can't find the container with id 1798fc45effa7f5a804aea00316c3a610725b293c9df80f302013a0aaff9e4a7
Sep 09 17:36:11.411229 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.411204    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"]
Sep 09 17:36:11.416111 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:36:11.416087    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4024d5_a487_491a_86fd_9f2cf7bca895.slice/crio-985b914a26a37da4d1196f695a35951e0a07c823f1982484200ae560cab8e6d6 WatchSource:0}: Error finding container 985b914a26a37da4d1196f695a35951e0a07c823f1982484200ae560cab8e6d6: Status 404 returned error can't find the container with id 985b914a26a37da4d1196f695a35951e0a07c823f1982484200ae560cab8e6d6
Sep 09 17:36:11.799610 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.799567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5" event={"ID":"7879e28f-0d6a-4713-a209-a40c46eabd56","Type":"ContainerStarted","Data":"42ed5c740ea8286c8bc0cb104ae8fcf73f0e080dcca5ada7954ca0e1eaf39aa8"}
Sep 09 17:36:11.800125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.799635    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:11.800125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.799661    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5" event={"ID":"7879e28f-0d6a-4713-a209-a40c46eabd56","Type":"ContainerStarted","Data":"1798fc45effa7f5a804aea00316c3a610725b293c9df80f302013a0aaff9e4a7"}
Sep 09 17:36:11.801423 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.801390    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt" event={"ID":"cb4024d5-a487-491a-86fd-9f2cf7bca895","Type":"ContainerStarted","Data":"f2b20b0570a2f78840bc2f31444761487bf4870148ff714b3edacb40bda3fa28"}
Sep 09 17:36:11.801543 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.801443    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt" event={"ID":"cb4024d5-a487-491a-86fd-9f2cf7bca895","Type":"ContainerStarted","Data":"985b914a26a37da4d1196f695a35951e0a07c823f1982484200ae560cab8e6d6"}
Sep 09 17:36:11.801543 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.801473    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:11.816945 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.816890    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5" podStartSLOduration=1.8168752609999999 podStartE2EDuration="1.816875261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:36:11.814991616 +0000 UTC m=+1147.443144316" watchObservedRunningTime="2026-09-09 17:36:11.816875261 +0000 UTC m=+1147.445027963"
Sep 09 17:36:11.835236 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:11.835179    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt" podStartSLOduration=1.8351599809999999 podStartE2EDuration="1.835159981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:36:11.834422987 +0000 UTC m=+1147.462575693" watchObservedRunningTime="2026-09-09 17:36:11.835159981 +0000 UTC m=+1147.463312708"
Sep 09 17:36:13.934581 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:13.934544    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"]
Sep 09 17:36:22.809033 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:22.808990    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:22.809538 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:22.809168    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:28.954696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:28.954583    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l" podUID="7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" containerName="main" containerID="cri-o://3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e" gracePeriod=45
Sep 09 17:36:29.198490 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.198463    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:36:29.308444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308360    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-dshm\") pod \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") "
Sep 09 17:36:29.308444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308402    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-model-cache\") pod \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") "
Sep 09 17:36:29.308444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308432    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-home\") pod \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") "
Sep 09 17:36:29.308701 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308509    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbknf\" (UniqueName: \"kubernetes.io/projected/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-kube-api-access-xbknf\") pod \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") "
Sep 09 17:36:29.308701 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308546    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tls-certs\") pod \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") "
Sep 09 17:36:29.308701 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308594    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tmp-dir\") pod \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\" (UID: \"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb\") "
Sep 09 17:36:29.308897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308694    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" (UID: "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:36:29.308897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308705    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-home" (OuterVolumeSpecName: "home") pod "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" (UID: "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:36:29.308897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308869    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" (UID: "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:36:29.309067 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.308983    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:29.309067 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.309000    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:29.309067 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.309009    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:29.310813 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.310786    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-dshm" (OuterVolumeSpecName: "dshm") pod "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" (UID: "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:36:29.310813 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.310807    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-kube-api-access-xbknf" (OuterVolumeSpecName: "kube-api-access-xbknf") pod "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" (UID: "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb"). InnerVolumeSpecName "kube-api-access-xbknf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:36:29.310993 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.310850    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" (UID: "7e9e3292-f98a-4128-b3b0-1d8cdb9833eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:36:29.410234 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.410197    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbknf\" (UniqueName: \"kubernetes.io/projected/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-kube-api-access-xbknf\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:29.410234 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.410228    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:29.410234 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.410239    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:29.904723 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.904694    2558 generic.go:356] "Generic (PLEG): container finished" podID="7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" containerID="3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e" exitCode=0
Sep 09 17:36:29.904913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.904804    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"
Sep 09 17:36:29.904913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.904797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l" event={"ID":"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb","Type":"ContainerDied","Data":"3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e"}
Sep 09 17:36:29.905048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.904923    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l" event={"ID":"7e9e3292-f98a-4128-b3b0-1d8cdb9833eb","Type":"ContainerDied","Data":"6eb26f18114632793ca34be8525f18c2f12ab1dc2732a0c39974a0a902dbb3bc"}
Sep 09 17:36:29.905048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.904945    2558 scope.go:117] "RemoveContainer" containerID="3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e"
Sep 09 17:36:29.914088 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.914067    2558 scope.go:117] "RemoveContainer" containerID="3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e"
Sep 09 17:36:29.914444 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:36:29.914421    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e\": container with ID starting with 3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e not found: ID does not exist" containerID="3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e"
Sep 09 17:36:29.914516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.914452    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e"} err="failed to get container status \"3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e\": rpc error: code = NotFound desc = could not find container \"3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e\": container with ID starting with 3f4297dfb0784c25bde1d08eed1b31a2afed245b944c73d60caf6c29a0a2b74e not found: ID does not exist"
Sep 09 17:36:29.928836 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.928787    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"]
Sep 09 17:36:29.931090 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:29.931071    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-37448c/autores-model-37448c-kserve-558fc4b6c9-qzj9l"]
Sep 09 17:36:30.817132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:30.817094    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" path="/var/lib/kubelet/pods/7e9e3292-f98a-4128-b3b0-1d8cdb9833eb/volumes"
Sep 09 17:36:33.900277 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.900227    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"]
Sep 09 17:36:33.900877 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.900751    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" containerName="main"
Sep 09 17:36:33.900877 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.900770    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" containerName="main"
Sep 09 17:36:33.901031 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.900879    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9e3292-f98a-4128-b3b0-1d8cdb9833eb" containerName="main"
Sep 09 17:36:33.903710 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.903688    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"
Sep 09 17:36:33.906210 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.906187    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:36:33.911364 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.911327    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"]
Sep 09 17:36:33.949085 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:33.949047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c\" (UID: \"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"
Sep 09 17:36:34.049930 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:34.049895    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c\" (UID: \"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"
Sep 09 17:36:34.050557 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:34.050536    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c\" (UID: \"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"
Sep 09 17:36:34.215897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:34.215862    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"
Sep 09 17:36:34.491198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:34.491173    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"]
Sep 09 17:36:34.493327 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:36:34.493299    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a70dcb_6e36_4fcc_81df_2f2e16ebd4af.slice/crio-bb5fa99dd436c26013adb01ef85efecae8eefc0a7325a396922077f2f1c65556 WatchSource:0}: Error finding container bb5fa99dd436c26013adb01ef85efecae8eefc0a7325a396922077f2f1c65556: Status 404 returned error can't find the container with id bb5fa99dd436c26013adb01ef85efecae8eefc0a7325a396922077f2f1c65556
Sep 09 17:36:34.937502 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:34.937471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c" event={"ID":"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af","Type":"ContainerStarted","Data":"5c3f41409b654494c5e1c85c47fc8db445d8acfb916b6d749ddb13698d8ce4b8"}
Sep 09 17:36:34.937873 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:34.937526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c" event={"ID":"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af","Type":"ContainerStarted","Data":"bb5fa99dd436c26013adb01ef85efecae8eefc0a7325a396922077f2f1c65556"}
Sep 09 17:36:34.953021 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:34.952966    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c" podStartSLOduration=1.952932165 podStartE2EDuration="1.952932165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:36:34.951368998 +0000 UTC m=+1170.579521701" watchObservedRunningTime="2026-09-09 17:36:34.952932165 +0000 UTC m=+1170.581084868"
Sep 09 17:36:35.944508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:35.944474    2558 generic.go:356] "Generic (PLEG): container finished" podID="d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af" containerID="5c3f41409b654494c5e1c85c47fc8db445d8acfb916b6d749ddb13698d8ce4b8" exitCode=0
Sep 09 17:36:35.944910 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:35.944513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c" event={"ID":"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af","Type":"ContainerDied","Data":"5c3f41409b654494c5e1c85c47fc8db445d8acfb916b6d749ddb13698d8ce4b8"}
Sep 09 17:36:37.080759 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.080729    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"]
Sep 09 17:36:37.081193 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.081175    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"
Sep 09 17:36:37.083877 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.083861    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.086166 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.086150    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-959540a1-kserve-self-signed-certs\""
Sep 09 17:36:37.091513 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.091488    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"]
Sep 09 17:36:37.179793 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.179753    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af-maas-api-service-ca\") pod \"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af\" (UID: \"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af\") "
Sep 09 17:36:37.180008 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.179983    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57mr\" (UniqueName: \"kubernetes.io/projected/e31fd649-bc60-417c-9c19-6c656876bf2b-kube-api-access-r57mr\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.180090 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.180051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-tmp-dir\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.180157 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.180102    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af" (UID: "d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:36:37.180157 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.180115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-home\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.180273 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.180155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-dshm\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.180273 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.180198    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-model-cache\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.180273 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.180226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e31fd649-bc60-417c-9c19-6c656876bf2b-tls-certs\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.180434 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.180306    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:37.281318 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57mr\" (UniqueName: \"kubernetes.io/projected/e31fd649-bc60-417c-9c19-6c656876bf2b-kube-api-access-r57mr\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281318 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-tmp-dir\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281318 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-home\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281583 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281337    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-dshm\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281583 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-model-cache\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281583 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e31fd649-bc60-417c-9c19-6c656876bf2b-tls-certs\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281814 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-home\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281854 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-tmp-dir\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.281891 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.281875    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-model-cache\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.283803 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.283775    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-dshm\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.283986 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.283949    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e31fd649-bc60-417c-9c19-6c656876bf2b-tls-certs\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.289399 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.289376    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57mr\" (UniqueName: \"kubernetes.io/projected/e31fd649-bc60-417c-9c19-6c656876bf2b-kube-api-access-r57mr\") pod \"e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") " pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.392566 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.392528    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:37.667501 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.667470    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"]
Sep 09 17:36:37.668695 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:36:37.668669    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode31fd649_bc60_417c_9c19_6c656876bf2b.slice/crio-70e5316692950d0b855429c8e09f7968b23a08380de99a719c3533cc07e7bcba WatchSource:0}: Error finding container 70e5316692950d0b855429c8e09f7968b23a08380de99a719c3533cc07e7bcba: Status 404 returned error can't find the container with id 70e5316692950d0b855429c8e09f7968b23a08380de99a719c3533cc07e7bcba
Sep 09 17:36:37.956796 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.956762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw" event={"ID":"e31fd649-bc60-417c-9c19-6c656876bf2b","Type":"ContainerStarted","Data":"678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a"}
Sep 09 17:36:37.957021 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.956807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw" event={"ID":"e31fd649-bc60-417c-9c19-6c656876bf2b","Type":"ContainerStarted","Data":"70e5316692950d0b855429c8e09f7968b23a08380de99a719c3533cc07e7bcba"}
Sep 09 17:36:37.958078 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.958058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c" event={"ID":"d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af","Type":"ContainerDied","Data":"bb5fa99dd436c26013adb01ef85efecae8eefc0a7325a396922077f2f1c65556"}
Sep 09 17:36:37.958175 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.958096    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"
Sep 09 17:36:37.958175 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.958107    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb5fa99dd436c26013adb01ef85efecae8eefc0a7325a396922077f2f1c65556"
Sep 09 17:36:37.980528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:37.980474    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw" podStartSLOduration=0.980458246 podStartE2EDuration="980.458246ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:36:37.979737425 +0000 UTC m=+1173.607890140" watchObservedRunningTime="2026-09-09 17:36:37.980458246 +0000 UTC m=+1173.608610972"
Sep 09 17:36:40.685180 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.685147    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"]
Sep 09 17:36:40.685608 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.685391    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt" podUID="cb4024d5-a487-491a-86fd-9f2cf7bca895" containerName="payload-processing" containerID="cri-o://f2b20b0570a2f78840bc2f31444761487bf4870148ff714b3edacb40bda3fa28" gracePeriod=30
Sep 09 17:36:40.698223 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.698190    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"]
Sep 09 17:36:40.698681 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.698629    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5" podUID="7879e28f-0d6a-4713-a209-a40c46eabd56" containerName="payload-pre-processing" containerID="cri-o://42ed5c740ea8286c8bc0cb104ae8fcf73f0e080dcca5ada7954ca0e1eaf39aa8" gracePeriod=30
Sep 09 17:36:40.848612 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.848561    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"]
Sep 09 17:36:40.853578 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.853545    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0aa81a54-4b66c"]
Sep 09 17:36:40.977694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.977662    2558 generic.go:356] "Generic (PLEG): container finished" podID="7879e28f-0d6a-4713-a209-a40c46eabd56" containerID="42ed5c740ea8286c8bc0cb104ae8fcf73f0e080dcca5ada7954ca0e1eaf39aa8" exitCode=0
Sep 09 17:36:40.977820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.977755    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5" event={"ID":"7879e28f-0d6a-4713-a209-a40c46eabd56","Type":"ContainerDied","Data":"42ed5c740ea8286c8bc0cb104ae8fcf73f0e080dcca5ada7954ca0e1eaf39aa8"}
Sep 09 17:36:40.979617 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.979596    2558 generic.go:356] "Generic (PLEG): container finished" podID="cb4024d5-a487-491a-86fd-9f2cf7bca895" containerID="f2b20b0570a2f78840bc2f31444761487bf4870148ff714b3edacb40bda3fa28" exitCode=0
Sep 09 17:36:40.979729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.979631    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt" event={"ID":"cb4024d5-a487-491a-86fd-9f2cf7bca895","Type":"ContainerDied","Data":"f2b20b0570a2f78840bc2f31444761487bf4870148ff714b3edacb40bda3fa28"}
Sep 09 17:36:40.979729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.979669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt" event={"ID":"cb4024d5-a487-491a-86fd-9f2cf7bca895","Type":"ContainerDied","Data":"985b914a26a37da4d1196f695a35951e0a07c823f1982484200ae560cab8e6d6"}
Sep 09 17:36:40.979729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.979688    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="985b914a26a37da4d1196f695a35951e0a07c823f1982484200ae560cab8e6d6"
Sep 09 17:36:40.981508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.981489    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:40.985809 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:40.985785    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:41.119175 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.119140    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h666l\" (UniqueName: \"kubernetes.io/projected/cb4024d5-a487-491a-86fd-9f2cf7bca895-kube-api-access-h666l\") pod \"cb4024d5-a487-491a-86fd-9f2cf7bca895\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") "
Sep 09 17:36:41.119345 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.119187    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e28f-0d6a-4713-a209-a40c46eabd56-plugins-config-volume\") pod \"7879e28f-0d6a-4713-a209-a40c46eabd56\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") "
Sep 09 17:36:41.119345 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.119274    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl8wc\" (UniqueName: \"kubernetes.io/projected/7879e28f-0d6a-4713-a209-a40c46eabd56-kube-api-access-nl8wc\") pod \"7879e28f-0d6a-4713-a209-a40c46eabd56\" (UID: \"7879e28f-0d6a-4713-a209-a40c46eabd56\") "
Sep 09 17:36:41.119345 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.119324    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4024d5-a487-491a-86fd-9f2cf7bca895-plugins-config-volume\") pod \"cb4024d5-a487-491a-86fd-9f2cf7bca895\" (UID: \"cb4024d5-a487-491a-86fd-9f2cf7bca895\") "
Sep 09 17:36:41.119669 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.119643    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7879e28f-0d6a-4713-a209-a40c46eabd56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7879e28f-0d6a-4713-a209-a40c46eabd56" (UID: "7879e28f-0d6a-4713-a209-a40c46eabd56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:36:41.119768 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.119746    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4024d5-a487-491a-86fd-9f2cf7bca895-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb4024d5-a487-491a-86fd-9f2cf7bca895" (UID: "cb4024d5-a487-491a-86fd-9f2cf7bca895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:36:41.121273 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.121252    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4024d5-a487-491a-86fd-9f2cf7bca895-kube-api-access-h666l" (OuterVolumeSpecName: "kube-api-access-h666l") pod "cb4024d5-a487-491a-86fd-9f2cf7bca895" (UID: "cb4024d5-a487-491a-86fd-9f2cf7bca895"). InnerVolumeSpecName "kube-api-access-h666l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:36:41.121376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.121359    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7879e28f-0d6a-4713-a209-a40c46eabd56-kube-api-access-nl8wc" (OuterVolumeSpecName: "kube-api-access-nl8wc") pod "7879e28f-0d6a-4713-a209-a40c46eabd56" (UID: "7879e28f-0d6a-4713-a209-a40c46eabd56"). InnerVolumeSpecName "kube-api-access-nl8wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:36:41.221054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.220974    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl8wc\" (UniqueName: \"kubernetes.io/projected/7879e28f-0d6a-4713-a209-a40c46eabd56-kube-api-access-nl8wc\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:41.221054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.221002    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4024d5-a487-491a-86fd-9f2cf7bca895-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:41.221054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.221013    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h666l\" (UniqueName: \"kubernetes.io/projected/cb4024d5-a487-491a-86fd-9f2cf7bca895-kube-api-access-h666l\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:41.221054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.221022    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7879e28f-0d6a-4713-a209-a40c46eabd56-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:36:41.987670 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.987585    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"
Sep 09 17:36:41.987670 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.987609    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5" event={"ID":"7879e28f-0d6a-4713-a209-a40c46eabd56","Type":"ContainerDied","Data":"1798fc45effa7f5a804aea00316c3a610725b293c9df80f302013a0aaff9e4a7"}
Sep 09 17:36:41.988140 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.987685    2558 scope.go:117] "RemoveContainer" containerID="42ed5c740ea8286c8bc0cb104ae8fcf73f0e080dcca5ada7954ca0e1eaf39aa8"
Sep 09 17:36:41.988140 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:41.987854    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"
Sep 09 17:36:42.013326 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:42.013293    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"]
Sep 09 17:36:42.015368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:42.015344    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0aa81a54-5f66f94f4f-447nt"]
Sep 09 17:36:42.026596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:42.026567    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"]
Sep 09 17:36:42.028558 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:42.028534    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aa81a54-69c99bbd9d-54xn5"]
Sep 09 17:36:42.818349 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:42.818306    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7879e28f-0d6a-4713-a209-a40c46eabd56" path="/var/lib/kubelet/pods/7879e28f-0d6a-4713-a209-a40c46eabd56/volumes"
Sep 09 17:36:42.818769 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:42.818749    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4024d5-a487-491a-86fd-9f2cf7bca895" path="/var/lib/kubelet/pods/cb4024d5-a487-491a-86fd-9f2cf7bca895/volumes"
Sep 09 17:36:42.819139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:42.819120    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af" path="/var/lib/kubelet/pods/d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af/volumes"
Sep 09 17:36:47.393128 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:47.393082    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:47.393590 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:47.393152    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:47.405342 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:47.405308    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:48.035849 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:48.035820    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:36:52.111824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:52.111783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"]
Sep 09 17:36:53.901372 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:36:53.901327    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"]
Sep 09 17:37:01.745199 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745159    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"]
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745482    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af" containerName="revoke-keys"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745493    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af" containerName="revoke-keys"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745515    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7879e28f-0d6a-4713-a209-a40c46eabd56" containerName="payload-pre-processing"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745521    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7879e28f-0d6a-4713-a209-a40c46eabd56" containerName="payload-pre-processing"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745533    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4024d5-a487-491a-86fd-9f2cf7bca895" containerName="payload-processing"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745539    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4024d5-a487-491a-86fd-9f2cf7bca895" containerName="payload-processing"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745587    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a70dcb-6e36-4fcc-81df-2f2e16ebd4af" containerName="revoke-keys"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745597    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7879e28f-0d6a-4713-a209-a40c46eabd56" containerName="payload-pre-processing"
Sep 09 17:37:01.745609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.745604    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4024d5-a487-491a-86fd-9f2cf7bca895" containerName="payload-processing"
Sep 09 17:37:01.750270 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.750250    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:01.752893 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.752866    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-95cd01\""
Sep 09 17:37:01.758159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.758136    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"]
Sep 09 17:37:01.806439 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.806405    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwmg\" (UniqueName: \"kubernetes.io/projected/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-kube-api-access-zfwmg\") pod \"payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:01.806604 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.806472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:01.907821 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.907784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwmg\" (UniqueName: \"kubernetes.io/projected/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-kube-api-access-zfwmg\") pod \"payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:01.908012 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.907853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:01.908603 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.908582    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:01.915595 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:01.915560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwmg\" (UniqueName: \"kubernetes.io/projected/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-kube-api-access-zfwmg\") pod \"payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:02.062469 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:02.062377    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:02.335453 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:02.335376    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"]
Sep 09 17:37:02.338693 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:37:02.338664    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f3b93f_beb1_4369_b8a6_f48a155cc8bb.slice/crio-767ccd5e39c5d462ad41ccd1518fcebaca59249857dd52eab6ff1111fba0d4ea WatchSource:0}: Error finding container 767ccd5e39c5d462ad41ccd1518fcebaca59249857dd52eab6ff1111fba0d4ea: Status 404 returned error can't find the container with id 767ccd5e39c5d462ad41ccd1518fcebaca59249857dd52eab6ff1111fba0d4ea
Sep 09 17:37:03.111025 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:03.110990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft" event={"ID":"27f3b93f-beb1-4369-b8a6-f48a155cc8bb","Type":"ContainerStarted","Data":"bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817"}
Sep 09 17:37:03.111517 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:03.111061    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft" event={"ID":"27f3b93f-beb1-4369-b8a6-f48a155cc8bb","Type":"ContainerStarted","Data":"767ccd5e39c5d462ad41ccd1518fcebaca59249857dd52eab6ff1111fba0d4ea"}
Sep 09 17:37:03.111517 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:03.111126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:03.130339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:03.130283    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft" podStartSLOduration=2.130262321 podStartE2EDuration="2.130262321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:37:03.127644922 +0000 UTC m=+1198.755797626" watchObservedRunningTime="2026-09-09 17:37:03.130262321 +0000 UTC m=+1198.758415027"
Sep 09 17:37:04.797801 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:04.797760    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:37:04.798303 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:04.798266    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:37:07.132891 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.132841    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw" podUID="e31fd649-bc60-417c-9c19-6c656876bf2b" containerName="main" containerID="cri-o://678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a" gracePeriod=45
Sep 09 17:37:07.386337 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.386263    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:37:07.450601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450569    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-home\") pod \"e31fd649-bc60-417c-9c19-6c656876bf2b\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") "
Sep 09 17:37:07.450601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450612    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e31fd649-bc60-417c-9c19-6c656876bf2b-tls-certs\") pod \"e31fd649-bc60-417c-9c19-6c656876bf2b\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") "
Sep 09 17:37:07.450860 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450644    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-tmp-dir\") pod \"e31fd649-bc60-417c-9c19-6c656876bf2b\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") "
Sep 09 17:37:07.450860 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450671    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-dshm\") pod \"e31fd649-bc60-417c-9c19-6c656876bf2b\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") "
Sep 09 17:37:07.450860 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450721    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r57mr\" (UniqueName: \"kubernetes.io/projected/e31fd649-bc60-417c-9c19-6c656876bf2b-kube-api-access-r57mr\") pod \"e31fd649-bc60-417c-9c19-6c656876bf2b\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") "
Sep 09 17:37:07.450860 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-model-cache\") pod \"e31fd649-bc60-417c-9c19-6c656876bf2b\" (UID: \"e31fd649-bc60-417c-9c19-6c656876bf2b\") "
Sep 09 17:37:07.451128 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450942    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e31fd649-bc60-417c-9c19-6c656876bf2b" (UID: "e31fd649-bc60-417c-9c19-6c656876bf2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:07.451128 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.450983    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-home" (OuterVolumeSpecName: "home") pod "e31fd649-bc60-417c-9c19-6c656876bf2b" (UID: "e31fd649-bc60-417c-9c19-6c656876bf2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:07.451233 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.451170    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:07.451233 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.451191    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:07.451233 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.451190    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e31fd649-bc60-417c-9c19-6c656876bf2b" (UID: "e31fd649-bc60-417c-9c19-6c656876bf2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:07.452924 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.452899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31fd649-bc60-417c-9c19-6c656876bf2b-kube-api-access-r57mr" (OuterVolumeSpecName: "kube-api-access-r57mr") pod "e31fd649-bc60-417c-9c19-6c656876bf2b" (UID: "e31fd649-bc60-417c-9c19-6c656876bf2b"). InnerVolumeSpecName "kube-api-access-r57mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:37:07.453209 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.453191    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31fd649-bc60-417c-9c19-6c656876bf2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e31fd649-bc60-417c-9c19-6c656876bf2b" (UID: "e31fd649-bc60-417c-9c19-6c656876bf2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:37:07.453364 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.453344    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-dshm" (OuterVolumeSpecName: "dshm") pod "e31fd649-bc60-417c-9c19-6c656876bf2b" (UID: "e31fd649-bc60-417c-9c19-6c656876bf2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:07.551639 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.551593    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:07.551639 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.551630    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e31fd649-bc60-417c-9c19-6c656876bf2b-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:07.551639 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.551643    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e31fd649-bc60-417c-9c19-6c656876bf2b-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:07.551885 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:07.551654    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r57mr\" (UniqueName: \"kubernetes.io/projected/e31fd649-bc60-417c-9c19-6c656876bf2b-kube-api-access-r57mr\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:08.140287 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.140250    2558 generic.go:356] "Generic (PLEG): container finished" podID="e31fd649-bc60-417c-9c19-6c656876bf2b" containerID="678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a" exitCode=0
Sep 09 17:37:08.140729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.140298    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"
Sep 09 17:37:08.140729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.140366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw" event={"ID":"e31fd649-bc60-417c-9c19-6c656876bf2b","Type":"ContainerDied","Data":"678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a"}
Sep 09 17:37:08.140729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.140410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw" event={"ID":"e31fd649-bc60-417c-9c19-6c656876bf2b","Type":"ContainerDied","Data":"70e5316692950d0b855429c8e09f7968b23a08380de99a719c3533cc07e7bcba"}
Sep 09 17:37:08.140729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.140433    2558 scope.go:117] "RemoveContainer" containerID="678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a"
Sep 09 17:37:08.150788 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.150769    2558 scope.go:117] "RemoveContainer" containerID="678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a"
Sep 09 17:37:08.151172 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:37:08.151151    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a\": container with ID starting with 678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a not found: ID does not exist" containerID="678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a"
Sep 09 17:37:08.151245 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.151185    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a"} err="failed to get container status \"678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a\": rpc error: code = NotFound desc = could not find container \"678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a\": container with ID starting with 678ca8d7a7441478bbdfb8abfa052900e309991a9c2df4a8f5760e41f4b29a1a not found: ID does not exist"
Sep 09 17:37:08.165827 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.165805    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"]
Sep 09 17:37:08.170937 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.170916    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-959540a1-kserve-558f85bbfc-zwdcw"]
Sep 09 17:37:08.816288 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.816255    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e31fd649-bc60-417c-9c19-6c656876bf2b" path="/var/lib/kubelet/pods/e31fd649-bc60-417c-9c19-6c656876bf2b/volumes"
Sep 09 17:37:08.923352 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:08.923317    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c" podUID="2ae472cc-4a6b-4d97-ba42-e745f19e0f06" containerName="main" containerID="cri-o://5558364c717d6ec9fd53b80baa6cea77ca60306ee6f4af4b93aebb84f9b73f83" gracePeriod=45
Sep 09 17:37:09.148694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.148665    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ae472cc-4a6b-4d97-ba42-e745f19e0f06" containerID="5558364c717d6ec9fd53b80baa6cea77ca60306ee6f4af4b93aebb84f9b73f83" exitCode=0
Sep 09 17:37:09.149067 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.148733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c" event={"ID":"2ae472cc-4a6b-4d97-ba42-e745f19e0f06","Type":"ContainerDied","Data":"5558364c717d6ec9fd53b80baa6cea77ca60306ee6f4af4b93aebb84f9b73f83"}
Sep 09 17:37:09.179415 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.179391    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:37:09.267627 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.267588    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tls-certs\") pod \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") "
Sep 09 17:37:09.267825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.267684    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-home\") pod \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") "
Sep 09 17:37:09.267825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.267704    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96znl\" (UniqueName: \"kubernetes.io/projected/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-kube-api-access-96znl\") pod \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") "
Sep 09 17:37:09.267825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.267721    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-model-cache\") pod \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") "
Sep 09 17:37:09.267825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.267771    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tmp-dir\") pod \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") "
Sep 09 17:37:09.267825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.267808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-dshm\") pod \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\" (UID: \"2ae472cc-4a6b-4d97-ba42-e745f19e0f06\") "
Sep 09 17:37:09.268121 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.268020    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ae472cc-4a6b-4d97-ba42-e745f19e0f06" (UID: "2ae472cc-4a6b-4d97-ba42-e745f19e0f06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:09.268121 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.268037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-home" (OuterVolumeSpecName: "home") pod "2ae472cc-4a6b-4d97-ba42-e745f19e0f06" (UID: "2ae472cc-4a6b-4d97-ba42-e745f19e0f06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:09.268226 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.268132    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:09.268226 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.268149    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:09.268226 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.268189    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ae472cc-4a6b-4d97-ba42-e745f19e0f06" (UID: "2ae472cc-4a6b-4d97-ba42-e745f19e0f06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:09.269767 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.269742    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-dshm" (OuterVolumeSpecName: "dshm") pod "2ae472cc-4a6b-4d97-ba42-e745f19e0f06" (UID: "2ae472cc-4a6b-4d97-ba42-e745f19e0f06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:37:09.269880 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.269814    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-kube-api-access-96znl" (OuterVolumeSpecName: "kube-api-access-96znl") pod "2ae472cc-4a6b-4d97-ba42-e745f19e0f06" (UID: "2ae472cc-4a6b-4d97-ba42-e745f19e0f06"). InnerVolumeSpecName "kube-api-access-96znl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:37:09.269880 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.269821    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ae472cc-4a6b-4d97-ba42-e745f19e0f06" (UID: "2ae472cc-4a6b-4d97-ba42-e745f19e0f06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:37:09.369483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.369396    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96znl\" (UniqueName: \"kubernetes.io/projected/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-kube-api-access-96znl\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:09.369483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.369427    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:09.369483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.369438    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:09.369483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:09.369446    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae472cc-4a6b-4d97-ba42-e745f19e0f06-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:37:10.155386 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:10.155353    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"
Sep 09 17:37:10.155784 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:10.155353    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c" event={"ID":"2ae472cc-4a6b-4d97-ba42-e745f19e0f06","Type":"ContainerDied","Data":"afe6b2b1a6d78aa7856cb9111747732f3459ce710046494187578bcb84912a9f"}
Sep 09 17:37:10.155784 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:10.155479    2558 scope.go:117] "RemoveContainer" containerID="5558364c717d6ec9fd53b80baa6cea77ca60306ee6f4af4b93aebb84f9b73f83"
Sep 09 17:37:10.177964 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:10.177927    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"]
Sep 09 17:37:10.185902 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:10.184440    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-959540a1-kserve-579454f675-jt99c"]
Sep 09 17:37:10.816727 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:10.816682    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae472cc-4a6b-4d97-ba42-e745f19e0f06" path="/var/lib/kubelet/pods/2ae472cc-4a6b-4d97-ba42-e745f19e0f06/volumes"
Sep 09 17:37:11.873741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.873706    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"]
Sep 09 17:37:11.874120 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.874044    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae472cc-4a6b-4d97-ba42-e745f19e0f06" containerName="main"
Sep 09 17:37:11.874120 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.874057    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae472cc-4a6b-4d97-ba42-e745f19e0f06" containerName="main"
Sep 09 17:37:11.874120 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.874065    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e31fd649-bc60-417c-9c19-6c656876bf2b" containerName="main"
Sep 09 17:37:11.874120 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.874070    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e31fd649-bc60-417c-9c19-6c656876bf2b" containerName="main"
Sep 09 17:37:11.874325 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.874136    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae472cc-4a6b-4d97-ba42-e745f19e0f06" containerName="main"
Sep 09 17:37:11.874325 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.874149    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e31fd649-bc60-417c-9c19-6c656876bf2b" containerName="main"
Sep 09 17:37:11.878291 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.878270    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:11.882621 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.882599    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-53da12\"/\"openshift-service-ca.crt\""
Sep 09 17:37:11.882621 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.882625    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-53da12\"/\"kube-root-ca.crt\""
Sep 09 17:37:11.882774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.882624    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-53da12\"/\"default-dockercfg-w62kx\""
Sep 09 17:37:11.882774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.882683    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-53da12\"/\"test-model-53da12-kserve-self-signed-certs\""
Sep 09 17:37:11.889163 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.889137    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"]
Sep 09 17:37:11.991661 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.991625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtgt\" (UniqueName: \"kubernetes.io/projected/d0524db3-b85a-4922-aebb-9a736bfe9d17-kube-api-access-xbtgt\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:11.991840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.991722    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-dshm\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:11.991840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.991770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-home\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:11.991840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.991793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0524db3-b85a-4922-aebb-9a736bfe9d17-tls-certs\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:11.991840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.991814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-model-cache\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:11.991840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:11.991828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-tmp-dir\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.092499 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.092456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtgt\" (UniqueName: \"kubernetes.io/projected/d0524db3-b85a-4922-aebb-9a736bfe9d17-kube-api-access-xbtgt\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.092700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.092517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-dshm\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.092700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.092549    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-home\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.092700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.092567    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0524db3-b85a-4922-aebb-9a736bfe9d17-tls-certs\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.092700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.092585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-model-cache\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.092700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.092606    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-tmp-dir\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.093007 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.092977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-home\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.093098 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.093024    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-tmp-dir\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.093098 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.093040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-model-cache\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.094853 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.094831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-dshm\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.095360 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.095342    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0524db3-b85a-4922-aebb-9a736bfe9d17-tls-certs\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.108378 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.108344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtgt\" (UniqueName: \"kubernetes.io/projected/d0524db3-b85a-4922-aebb-9a736bfe9d17-kube-api-access-xbtgt\") pod \"test-model-53da12-kserve-555bd8bccc-s5zf2\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") " pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.186763 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.186728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:12.460475 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:12.460446    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"]
Sep 09 17:37:12.461792 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:37:12.461767    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0524db3_b85a_4922_aebb_9a736bfe9d17.slice/crio-7c855e8cb26c8cf1288e5827dcd4c76ea997b5b2137456ac51abac16362e78af WatchSource:0}: Error finding container 7c855e8cb26c8cf1288e5827dcd4c76ea997b5b2137456ac51abac16362e78af: Status 404 returned error can't find the container with id 7c855e8cb26c8cf1288e5827dcd4c76ea997b5b2137456ac51abac16362e78af
Sep 09 17:37:13.174396 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:13.174357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2" event={"ID":"d0524db3-b85a-4922-aebb-9a736bfe9d17","Type":"ContainerStarted","Data":"14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a"}
Sep 09 17:37:13.174396 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:13.174403    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2" event={"ID":"d0524db3-b85a-4922-aebb-9a736bfe9d17","Type":"ContainerStarted","Data":"7c855e8cb26c8cf1288e5827dcd4c76ea997b5b2137456ac51abac16362e78af"}
Sep 09 17:37:13.192376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:13.192322    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2" podStartSLOduration=2.192306178 podStartE2EDuration="2.192306178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:37:13.19179953 +0000 UTC m=+1208.819952234" watchObservedRunningTime="2026-09-09 17:37:13.192306178 +0000 UTC m=+1208.820458880"
Sep 09 17:37:14.118938 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:14.118908    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:37:22.187054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:22.187006    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:22.187473 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:22.187069    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:22.199794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:22.199765    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:22.239289 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:22.239261    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:37:46.694433 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.694389    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"]
Sep 09 17:37:46.698081 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.698056    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.700965 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.700927    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-95cd01\"/\"openshift-service-ca.crt\""
Sep 09 17:37:46.702005 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.701984    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-95cd01\"/\"default-dockercfg-z97rd\""
Sep 09 17:37:46.702149 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.702086    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-95cd01\"/\"kube-root-ca.crt\""
Sep 09 17:37:46.702149 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.702124    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-95cd01\"/\"explicit-model-95cd01-kserve-self-signed-certs\""
Sep 09 17:37:46.707704 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.707684    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"]
Sep 09 17:37:46.815193 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.815159    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-home\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.815391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.815281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tls-certs\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.815391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.815319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-model-cache\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.815521 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.815391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-dshm\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.815521 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.815415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tmp-dir\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.815521 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.815443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6fr\" (UniqueName: \"kubernetes.io/projected/c81d7673-cfd5-423a-8e8a-0820c0f2227b-kube-api-access-vb6fr\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.916676 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.916641    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tls-certs\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.916802 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.916682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-model-cache\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.916802 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.916747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-dshm\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.916802 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.916772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tmp-dir\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.916802 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.916796    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6fr\" (UniqueName: \"kubernetes.io/projected/c81d7673-cfd5-423a-8e8a-0820c0f2227b-kube-api-access-vb6fr\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.917026 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.916831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-home\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.917254 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.917223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tmp-dir\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.917379 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.917276    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-home\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.917448 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.917402    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-model-cache\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.919146 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.919114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-dshm\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.919382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.919365    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tls-certs\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:46.925487 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:46.925456    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6fr\" (UniqueName: \"kubernetes.io/projected/c81d7673-cfd5-423a-8e8a-0820c0f2227b-kube-api-access-vb6fr\") pod \"explicit-model-95cd01-kserve-5f9947845c-csxl2\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") " pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:47.008796 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:47.008713    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:47.284400 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:47.284364    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"]
Sep 09 17:37:47.285878 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:37:47.285851    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc81d7673_cfd5_423a_8e8a_0820c0f2227b.slice/crio-3594d2a9f6acead17fb1a9f358e186e7fbfbf96771520e5d656298c11616e2b4 WatchSource:0}: Error finding container 3594d2a9f6acead17fb1a9f358e186e7fbfbf96771520e5d656298c11616e2b4: Status 404 returned error can't find the container with id 3594d2a9f6acead17fb1a9f358e186e7fbfbf96771520e5d656298c11616e2b4
Sep 09 17:37:47.362486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:47.362450    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2" event={"ID":"c81d7673-cfd5-423a-8e8a-0820c0f2227b","Type":"ContainerStarted","Data":"4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241"}
Sep 09 17:37:47.362486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:47.362492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2" event={"ID":"c81d7673-cfd5-423a-8e8a-0820c0f2227b","Type":"ContainerStarted","Data":"3594d2a9f6acead17fb1a9f358e186e7fbfbf96771520e5d656298c11616e2b4"}
Sep 09 17:37:47.381729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:47.381671    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2" podStartSLOduration=1.381652699 podStartE2EDuration="1.381652699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:37:47.379039541 +0000 UTC m=+1243.007192256" watchObservedRunningTime="2026-09-09 17:37:47.381652699 +0000 UTC m=+1243.009805470"
Sep 09 17:37:57.008899 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:57.008853    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:57.009525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:57.008916    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:57.027638 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:57.027602    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:37:57.431777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:37:57.431749    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:38:13.874611 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:13.874579    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"]
Sep 09 17:38:13.998725 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:13.998688    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"]
Sep 09 17:38:14.003149 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.003131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"
Sep 09 17:38:14.006171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.006114    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:38:14.009175 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.009151    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"]
Sep 09 17:38:14.058312 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.058280    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a18e43-96a5-4231-a8a4-d3dc690648f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-95cd01-4bd96\" (UID: \"08a18e43-96a5-4231-a8a4-d3dc690648f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"
Sep 09 17:38:14.159771 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.159678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a18e43-96a5-4231-a8a4-d3dc690648f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-95cd01-4bd96\" (UID: \"08a18e43-96a5-4231-a8a4-d3dc690648f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"
Sep 09 17:38:14.160391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.160344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a18e43-96a5-4231-a8a4-d3dc690648f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-95cd01-4bd96\" (UID: \"08a18e43-96a5-4231-a8a4-d3dc690648f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"
Sep 09 17:38:14.316710 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.316678    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"
Sep 09 17:38:14.594588 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:14.594555    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"]
Sep 09 17:38:14.596184 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:38:14.596148    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a18e43_96a5_4231_a8a4_d3dc690648f4.slice/crio-717c1de4512650f3f93b939afaedb103eb6cc3bc02b452bb77c3ecebdf2941d8 WatchSource:0}: Error finding container 717c1de4512650f3f93b939afaedb103eb6cc3bc02b452bb77c3ecebdf2941d8: Status 404 returned error can't find the container with id 717c1de4512650f3f93b939afaedb103eb6cc3bc02b452bb77c3ecebdf2941d8
Sep 09 17:38:15.524924 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:15.524886    2558 generic.go:356] "Generic (PLEG): container finished" podID="08a18e43-96a5-4231-a8a4-d3dc690648f4" containerID="3d1d318b9e4505fba2990ea52938b64d290c508ce07ed380b48d43105189e6fd" exitCode=0
Sep 09 17:38:15.525332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:15.524930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96" event={"ID":"08a18e43-96a5-4231-a8a4-d3dc690648f4","Type":"ContainerDied","Data":"3d1d318b9e4505fba2990ea52938b64d290c508ce07ed380b48d43105189e6fd"}
Sep 09 17:38:15.525332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:15.524996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96" event={"ID":"08a18e43-96a5-4231-a8a4-d3dc690648f4","Type":"ContainerStarted","Data":"717c1de4512650f3f93b939afaedb103eb6cc3bc02b452bb77c3ecebdf2941d8"}
Sep 09 17:38:16.659285 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:16.659259    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"
Sep 09 17:38:16.783297 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:16.783202    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a18e43-96a5-4231-a8a4-d3dc690648f4-maas-api-service-ca\") pod \"08a18e43-96a5-4231-a8a4-d3dc690648f4\" (UID: \"08a18e43-96a5-4231-a8a4-d3dc690648f4\") "
Sep 09 17:38:16.783542 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:16.783518    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a18e43-96a5-4231-a8a4-d3dc690648f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08a18e43-96a5-4231-a8a4-d3dc690648f4" (UID: "08a18e43-96a5-4231-a8a4-d3dc690648f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:38:16.884471 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:16.884432    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a18e43-96a5-4231-a8a4-d3dc690648f4-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:17.537024 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:17.536988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96" event={"ID":"08a18e43-96a5-4231-a8a4-d3dc690648f4","Type":"ContainerDied","Data":"717c1de4512650f3f93b939afaedb103eb6cc3bc02b452bb77c3ecebdf2941d8"}
Sep 09 17:38:17.537205 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:17.537042    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="717c1de4512650f3f93b939afaedb103eb6cc3bc02b452bb77c3ecebdf2941d8"
Sep 09 17:38:17.537205 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:17.537006    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"
Sep 09 17:38:21.090136 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.090105    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"]
Sep 09 17:38:21.090485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.090422    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a18e43-96a5-4231-a8a4-d3dc690648f4" containerName="revoke-keys"
Sep 09 17:38:21.090485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.090433    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a18e43-96a5-4231-a8a4-d3dc690648f4" containerName="revoke-keys"
Sep 09 17:38:21.090558 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.090501    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a18e43-96a5-4231-a8a4-d3dc690648f4" containerName="revoke-keys"
Sep 09 17:38:21.093745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.093725    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.096518 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.096499    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-71f58f92\""
Sep 09 17:38:21.096820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.096805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-71f58f92-dockercfg-bp8tg\""
Sep 09 17:38:21.107632 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.107606    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"]
Sep 09 17:38:21.124168 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.124133    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"]
Sep 09 17:38:21.127427 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.127404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.136434 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.136406    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"]
Sep 09 17:38:21.224059 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.224024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78131cdf-9a89-41a1-bb75-33833866737b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.224059 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.224071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") " pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.224326 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.224162    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfslk\" (UniqueName: \"kubernetes.io/projected/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-kube-api-access-qfslk\") pod \"payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") " pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.224326 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.224227    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2l76\" (UniqueName: \"kubernetes.io/projected/78131cdf-9a89-41a1-bb75-33833866737b-kube-api-access-p2l76\") pod \"payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.325626 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.325589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78131cdf-9a89-41a1-bb75-33833866737b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.325626 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.325629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") " pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.325828 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.325672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfslk\" (UniqueName: \"kubernetes.io/projected/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-kube-api-access-qfslk\") pod \"payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") " pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.325828 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.325719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2l76\" (UniqueName: \"kubernetes.io/projected/78131cdf-9a89-41a1-bb75-33833866737b-kube-api-access-p2l76\") pod \"payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.326358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.326336    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") " pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.326410 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.326346    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78131cdf-9a89-41a1-bb75-33833866737b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.333734 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.333705    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfslk\" (UniqueName: \"kubernetes.io/projected/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-kube-api-access-qfslk\") pod \"payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") " pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.333866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.333840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2l76\" (UniqueName: \"kubernetes.io/projected/78131cdf-9a89-41a1-bb75-33833866737b-kube-api-access-p2l76\") pod \"payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.403904 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.403806    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:21.439095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.439061    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:21.683167 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.683134    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"]
Sep 09 17:38:21.684316 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:38:21.684285    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78131cdf_9a89_41a1_bb75_33833866737b.slice/crio-ba64eea79d5646fa6ec364659776ab156186040f44ed295a91e2a5cb9a2b7ede WatchSource:0}: Error finding container ba64eea79d5646fa6ec364659776ab156186040f44ed295a91e2a5cb9a2b7ede: Status 404 returned error can't find the container with id ba64eea79d5646fa6ec364659776ab156186040f44ed295a91e2a5cb9a2b7ede
Sep 09 17:38:21.726489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:21.726460    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"]
Sep 09 17:38:21.740503 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:38:21.740474    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8ce7ed_5d44_420a_aec5_132e004f7bb3.slice/crio-b0c813166e4649a6ca8b64ca25a63dd06b0f34f56bb4fb963574da3d9125c654 WatchSource:0}: Error finding container b0c813166e4649a6ca8b64ca25a63dd06b0f34f56bb4fb963574da3d9125c654: Status 404 returned error can't find the container with id b0c813166e4649a6ca8b64ca25a63dd06b0f34f56bb4fb963574da3d9125c654
Sep 09 17:38:22.573548 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.573504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx" event={"ID":"78131cdf-9a89-41a1-bb75-33833866737b","Type":"ContainerStarted","Data":"db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37"}
Sep 09 17:38:22.574015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.573572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx" event={"ID":"78131cdf-9a89-41a1-bb75-33833866737b","Type":"ContainerStarted","Data":"ba64eea79d5646fa6ec364659776ab156186040f44ed295a91e2a5cb9a2b7ede"}
Sep 09 17:38:22.574015 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.573602    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:22.575392 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.575370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5" event={"ID":"3d8ce7ed-5d44-420a-aec5-132e004f7bb3","Type":"ContainerStarted","Data":"f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f"}
Sep 09 17:38:22.575464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.575410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5" event={"ID":"3d8ce7ed-5d44-420a-aec5-132e004f7bb3","Type":"ContainerStarted","Data":"b0c813166e4649a6ca8b64ca25a63dd06b0f34f56bb4fb963574da3d9125c654"}
Sep 09 17:38:22.575464 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.575429    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:22.590630 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.590587    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx" podStartSLOduration=1.590573376 podStartE2EDuration="1.590573376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:38:22.589606581 +0000 UTC m=+1278.217759284" watchObservedRunningTime="2026-09-09 17:38:22.590573376 +0000 UTC m=+1278.218726079"
Sep 09 17:38:22.608500 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:22.608457    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5" podStartSLOduration=1.608443865 podStartE2EDuration="1.608443865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:38:22.60664935 +0000 UTC m=+1278.234802084" watchObservedRunningTime="2026-09-09 17:38:22.608443865 +0000 UTC m=+1278.236596568"
Sep 09 17:38:24.724291 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:24.724250    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"]
Sep 09 17:38:24.724751 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:24.724537    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft" podUID="27f3b93f-beb1-4369-b8a6-f48a155cc8bb" containerName="payload-pre-processing" containerID="cri-o://bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817" gracePeriod=30
Sep 09 17:38:24.880727 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:24.880693    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"]
Sep 09 17:38:24.883774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:24.883742    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-95cd01-4bd96"]
Sep 09 17:38:24.990416 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:24.990353    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:38:25.061163 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.061130    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfwmg\" (UniqueName: \"kubernetes.io/projected/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-kube-api-access-zfwmg\") pod \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") "
Sep 09 17:38:25.061368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.061351    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-plugins-config-volume\") pod \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\" (UID: \"27f3b93f-beb1-4369-b8a6-f48a155cc8bb\") "
Sep 09 17:38:25.061701 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.061677    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27f3b93f-beb1-4369-b8a6-f48a155cc8bb" (UID: "27f3b93f-beb1-4369-b8a6-f48a155cc8bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:38:25.063512 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.063486    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-kube-api-access-zfwmg" (OuterVolumeSpecName: "kube-api-access-zfwmg") pod "27f3b93f-beb1-4369-b8a6-f48a155cc8bb" (UID: "27f3b93f-beb1-4369-b8a6-f48a155cc8bb"). InnerVolumeSpecName "kube-api-access-zfwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:38:25.162601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.162564    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfwmg\" (UniqueName: \"kubernetes.io/projected/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-kube-api-access-zfwmg\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:25.162601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.162597    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f3b93f-beb1-4369-b8a6-f48a155cc8bb-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:25.594007 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.593939    2558 generic.go:356] "Generic (PLEG): container finished" podID="27f3b93f-beb1-4369-b8a6-f48a155cc8bb" containerID="bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817" exitCode=0
Sep 09 17:38:25.594007 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.594011    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"
Sep 09 17:38:25.594282 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.594037    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft" event={"ID":"27f3b93f-beb1-4369-b8a6-f48a155cc8bb","Type":"ContainerDied","Data":"bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817"}
Sep 09 17:38:25.594282 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.594094    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft" event={"ID":"27f3b93f-beb1-4369-b8a6-f48a155cc8bb","Type":"ContainerDied","Data":"767ccd5e39c5d462ad41ccd1518fcebaca59249857dd52eab6ff1111fba0d4ea"}
Sep 09 17:38:25.594282 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.594119    2558 scope.go:117] "RemoveContainer" containerID="bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817"
Sep 09 17:38:25.603265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.603246    2558 scope.go:117] "RemoveContainer" containerID="bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817"
Sep 09 17:38:25.603558 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:38:25.603537    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817\": container with ID starting with bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817 not found: ID does not exist" containerID="bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817"
Sep 09 17:38:25.603610 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.603566    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817"} err="failed to get container status \"bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817\": rpc error: code = NotFound desc = could not find container \"bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817\": container with ID starting with bc00804850d133b3d6aa72e3268133b0cc63e67e830bbd4de9be6978a9d2b817 not found: ID does not exist"
Sep 09 17:38:25.615814 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.615783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"]
Sep 09 17:38:25.620777 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:25.620752    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-95cd01-655bbd475d-dt7ft"]
Sep 09 17:38:26.816080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:26.816045    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a18e43-96a5-4231-a8a4-d3dc690648f4" path="/var/lib/kubelet/pods/08a18e43-96a5-4231-a8a4-d3dc690648f4/volumes"
Sep 09 17:38:26.816458 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:26.816412    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f3b93f-beb1-4369-b8a6-f48a155cc8bb" path="/var/lib/kubelet/pods/27f3b93f-beb1-4369-b8a6-f48a155cc8bb/volumes"
Sep 09 17:38:28.894257 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:28.894210    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2" podUID="c81d7673-cfd5-423a-8e8a-0820c0f2227b" containerName="main" containerID="cri-o://4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241" gracePeriod=45
Sep 09 17:38:29.132653 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.132625    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:38:29.301188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301148    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-model-cache\") pod \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") "
Sep 09 17:38:29.301397 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301203    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb6fr\" (UniqueName: \"kubernetes.io/projected/c81d7673-cfd5-423a-8e8a-0820c0f2227b-kube-api-access-vb6fr\") pod \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") "
Sep 09 17:38:29.301397 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301266    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-home\") pod \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") "
Sep 09 17:38:29.301397 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301317    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-dshm\") pod \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") "
Sep 09 17:38:29.301555 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301412    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tls-certs\") pod \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") "
Sep 09 17:38:29.301555 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301441    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tmp-dir\") pod \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\" (UID: \"c81d7673-cfd5-423a-8e8a-0820c0f2227b\") "
Sep 09 17:38:29.301555 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301464    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c81d7673-cfd5-423a-8e8a-0820c0f2227b" (UID: "c81d7673-cfd5-423a-8e8a-0820c0f2227b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:29.301744 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301611    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-home" (OuterVolumeSpecName: "home") pod "c81d7673-cfd5-423a-8e8a-0820c0f2227b" (UID: "c81d7673-cfd5-423a-8e8a-0820c0f2227b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:29.301804 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301739    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:29.301804 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301759    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:29.301804 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.301779    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c81d7673-cfd5-423a-8e8a-0820c0f2227b" (UID: "c81d7673-cfd5-423a-8e8a-0820c0f2227b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:29.303586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.303550    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c81d7673-cfd5-423a-8e8a-0820c0f2227b" (UID: "c81d7673-cfd5-423a-8e8a-0820c0f2227b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:38:29.303699 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.303587    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-dshm" (OuterVolumeSpecName: "dshm") pod "c81d7673-cfd5-423a-8e8a-0820c0f2227b" (UID: "c81d7673-cfd5-423a-8e8a-0820c0f2227b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:29.303699 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.303611    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81d7673-cfd5-423a-8e8a-0820c0f2227b-kube-api-access-vb6fr" (OuterVolumeSpecName: "kube-api-access-vb6fr") pod "c81d7673-cfd5-423a-8e8a-0820c0f2227b" (UID: "c81d7673-cfd5-423a-8e8a-0820c0f2227b"). InnerVolumeSpecName "kube-api-access-vb6fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:38:29.402792 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.402754    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:29.402792 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.402782    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:29.402792 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.402791    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6fr\" (UniqueName: \"kubernetes.io/projected/c81d7673-cfd5-423a-8e8a-0820c0f2227b-kube-api-access-vb6fr\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:29.402792 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.402800    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81d7673-cfd5-423a-8e8a-0820c0f2227b-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:29.618942 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.618849    2558 generic.go:356] "Generic (PLEG): container finished" podID="c81d7673-cfd5-423a-8e8a-0820c0f2227b" containerID="4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241" exitCode=0
Sep 09 17:38:29.618942 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.618892    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"
Sep 09 17:38:29.619139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.618940    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2" event={"ID":"c81d7673-cfd5-423a-8e8a-0820c0f2227b","Type":"ContainerDied","Data":"4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241"}
Sep 09 17:38:29.619139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.619011    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2" event={"ID":"c81d7673-cfd5-423a-8e8a-0820c0f2227b","Type":"ContainerDied","Data":"3594d2a9f6acead17fb1a9f358e186e7fbfbf96771520e5d656298c11616e2b4"}
Sep 09 17:38:29.619139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.619032    2558 scope.go:117] "RemoveContainer" containerID="4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241"
Sep 09 17:38:29.628299 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.628284    2558 scope.go:117] "RemoveContainer" containerID="4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241"
Sep 09 17:38:29.628560 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:38:29.628543    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241\": container with ID starting with 4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241 not found: ID does not exist" containerID="4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241"
Sep 09 17:38:29.628608 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.628568    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241"} err="failed to get container status \"4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241\": rpc error: code = NotFound desc = could not find container \"4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241\": container with ID starting with 4e7b1dd42e62362e22fc5daefd3de550e6182bb46eaed47da7f329bf2d3d2241 not found: ID does not exist"
Sep 09 17:38:29.640303 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.640279    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"]
Sep 09 17:38:29.642297 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:29.642277    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-95cd01/explicit-model-95cd01-kserve-5f9947845c-csxl2"]
Sep 09 17:38:30.816445 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:30.816410    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c81d7673-cfd5-423a-8e8a-0820c0f2227b" path="/var/lib/kubelet/pods/c81d7673-cfd5-423a-8e8a-0820c0f2227b/volumes"
Sep 09 17:38:30.886868 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:30.886837    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"]
Sep 09 17:38:32.103429 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.103394    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"]
Sep 09 17:38:32.227604 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.227568    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"]
Sep 09 17:38:32.227914 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.227901    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81d7673-cfd5-423a-8e8a-0820c0f2227b" containerName="main"
Sep 09 17:38:32.227990 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.227916    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81d7673-cfd5-423a-8e8a-0820c0f2227b" containerName="main"
Sep 09 17:38:32.227990 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.227929    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f3b93f-beb1-4369-b8a6-f48a155cc8bb" containerName="payload-pre-processing"
Sep 09 17:38:32.227990 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.227935    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f3b93f-beb1-4369-b8a6-f48a155cc8bb" containerName="payload-pre-processing"
Sep 09 17:38:32.228097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.228013    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f3b93f-beb1-4369-b8a6-f48a155cc8bb" containerName="payload-pre-processing"
Sep 09 17:38:32.228097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.228023    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81d7673-cfd5-423a-8e8a-0820c0f2227b" containerName="main"
Sep 09 17:38:32.232163 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.232140    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"
Sep 09 17:38:32.235126 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.235074    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:38:32.237613 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.237551    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"]
Sep 09 17:38:32.327627 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.327588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9414f1-3920-489f-b3b6-c0f37c9f445f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv\" (UID: \"ba9414f1-3920-489f-b3b6-c0f37c9f445f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"
Sep 09 17:38:32.428724 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.428691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9414f1-3920-489f-b3b6-c0f37c9f445f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv\" (UID: \"ba9414f1-3920-489f-b3b6-c0f37c9f445f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"
Sep 09 17:38:32.429394 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.429374    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9414f1-3920-489f-b3b6-c0f37c9f445f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv\" (UID: \"ba9414f1-3920-489f-b3b6-c0f37c9f445f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"
Sep 09 17:38:32.545140 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.545101    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"
Sep 09 17:38:32.818996 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:32.818972    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"]
Sep 09 17:38:32.820839 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:38:32.820810    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9414f1_3920_489f_b3b6_c0f37c9f445f.slice/crio-fa7e63911e1bad5a360a43eb2435d5cc907fb04a6e15089f020db7e33032ef6d WatchSource:0}: Error finding container fa7e63911e1bad5a360a43eb2435d5cc907fb04a6e15089f020db7e33032ef6d: Status 404 returned error can't find the container with id fa7e63911e1bad5a360a43eb2435d5cc907fb04a6e15089f020db7e33032ef6d
Sep 09 17:38:33.584478 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:33.584442    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:38:33.584887 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:33.584515    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:38:33.647035 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:33.646999    2558 generic.go:356] "Generic (PLEG): container finished" podID="ba9414f1-3920-489f-b3b6-c0f37c9f445f" containerID="2caf5975a54f076723f1e9302708c51449de5c24e9e776223488382937e9328b" exitCode=0
Sep 09 17:38:33.647189 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:33.647093    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv" event={"ID":"ba9414f1-3920-489f-b3b6-c0f37c9f445f","Type":"ContainerDied","Data":"2caf5975a54f076723f1e9302708c51449de5c24e9e776223488382937e9328b"}
Sep 09 17:38:33.647189 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:33.647150    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv" event={"ID":"ba9414f1-3920-489f-b3b6-c0f37c9f445f","Type":"ContainerStarted","Data":"fa7e63911e1bad5a360a43eb2435d5cc907fb04a6e15089f020db7e33032ef6d"}
Sep 09 17:38:34.788902 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:34.788876    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"
Sep 09 17:38:34.952876 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:34.952847    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9414f1-3920-489f-b3b6-c0f37c9f445f-maas-api-service-ca\") pod \"ba9414f1-3920-489f-b3b6-c0f37c9f445f\" (UID: \"ba9414f1-3920-489f-b3b6-c0f37c9f445f\") "
Sep 09 17:38:34.953248 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:34.953224    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9414f1-3920-489f-b3b6-c0f37c9f445f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba9414f1-3920-489f-b3b6-c0f37c9f445f" (UID: "ba9414f1-3920-489f-b3b6-c0f37c9f445f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:38:35.053508 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:35.053476    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9414f1-3920-489f-b3b6-c0f37c9f445f-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:35.659824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:35.659789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv" event={"ID":"ba9414f1-3920-489f-b3b6-c0f37c9f445f","Type":"ContainerDied","Data":"fa7e63911e1bad5a360a43eb2435d5cc907fb04a6e15089f020db7e33032ef6d"}
Sep 09 17:38:35.659824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:35.659823    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"
Sep 09 17:38:35.659824 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:35.659833    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa7e63911e1bad5a360a43eb2435d5cc907fb04a6e15089f020db7e33032ef6d"
Sep 09 17:38:42.497845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.497805    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"]
Sep 09 17:38:42.498884 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.498086    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q" podUID="7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" containerName="payload-processing" containerID="cri-o://c8baf1ebc71dd230f2afca24e4da173719cf32bde5075ba1aa9d9937a8cab1f7" gracePeriod=30
Sep 09 17:38:42.708462 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.708423    2558 generic.go:356] "Generic (PLEG): container finished" podID="7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" containerID="c8baf1ebc71dd230f2afca24e4da173719cf32bde5075ba1aa9d9937a8cab1f7" exitCode=0
Sep 09 17:38:42.708650 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.708500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q" event={"ID":"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c","Type":"ContainerDied","Data":"c8baf1ebc71dd230f2afca24e4da173719cf32bde5075ba1aa9d9937a8cab1f7"}
Sep 09 17:38:42.757947 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.757876    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:38:42.808493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.808462    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-plugins-config-volume\") pod \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") "
Sep 09 17:38:42.808667 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.808546    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b2xj\" (UniqueName: \"kubernetes.io/projected/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-kube-api-access-6b2xj\") pod \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\" (UID: \"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c\") "
Sep 09 17:38:42.808899 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.808876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" (UID: "7f7aee36-49c8-4c6c-a7a3-538aa314ed8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:38:42.810736 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.810708    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-kube-api-access-6b2xj" (OuterVolumeSpecName: "kube-api-access-6b2xj") pod "7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" (UID: "7f7aee36-49c8-4c6c-a7a3-538aa314ed8c"). InnerVolumeSpecName "kube-api-access-6b2xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:38:42.817502 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.817482    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"]
Sep 09 17:38:42.819181 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.819162    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea996c-zcgjv"]
Sep 09 17:38:42.909619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.909582    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:42.909619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:42.909613    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b2xj\" (UniqueName: \"kubernetes.io/projected/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c-kube-api-access-6b2xj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:43.715039 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:43.715010    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"
Sep 09 17:38:43.715527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:43.715012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q" event={"ID":"7f7aee36-49c8-4c6c-a7a3-538aa314ed8c","Type":"ContainerDied","Data":"48bfab409cb8d9a88c07eedf80847367cf94b0239036a1ab92d62a43b78f9733"}
Sep 09 17:38:43.715527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:43.715140    2558 scope.go:117] "RemoveContainer" containerID="c8baf1ebc71dd230f2afca24e4da173719cf32bde5075ba1aa9d9937a8cab1f7"
Sep 09 17:38:43.735079 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:43.735041    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"]
Sep 09 17:38:43.740269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:43.740243    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ea996c-78cc5fff55-5nm7q"]
Sep 09 17:38:44.817161 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:44.817128    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" path="/var/lib/kubelet/pods/7f7aee36-49c8-4c6c-a7a3-538aa314ed8c/volumes"
Sep 09 17:38:44.817555 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:44.817452    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba9414f1-3920-489f-b3b6-c0f37c9f445f" path="/var/lib/kubelet/pods/ba9414f1-3920-489f-b3b6-c0f37c9f445f/volumes"
Sep 09 17:38:45.910717 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:45.910674    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd" podUID="31430376-c599-4c28-bc72-ffcb86b7da9c" containerName="main" containerID="cri-o://fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc" gracePeriod=45
Sep 09 17:38:46.156985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.156949    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:38:46.237791 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.237749    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-model-cache\") pod \"31430376-c599-4c28-bc72-ffcb86b7da9c\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") "
Sep 09 17:38:46.238028 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.237804    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ggrp\" (UniqueName: \"kubernetes.io/projected/31430376-c599-4c28-bc72-ffcb86b7da9c-kube-api-access-2ggrp\") pod \"31430376-c599-4c28-bc72-ffcb86b7da9c\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") "
Sep 09 17:38:46.238028 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.237860    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-dshm\") pod \"31430376-c599-4c28-bc72-ffcb86b7da9c\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") "
Sep 09 17:38:46.238028 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.237894    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-tmp-dir\") pod \"31430376-c599-4c28-bc72-ffcb86b7da9c\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") "
Sep 09 17:38:46.238028 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.237945    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31430376-c599-4c28-bc72-ffcb86b7da9c-tls-certs\") pod \"31430376-c599-4c28-bc72-ffcb86b7da9c\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") "
Sep 09 17:38:46.238028 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.238002    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-home\") pod \"31430376-c599-4c28-bc72-ffcb86b7da9c\" (UID: \"31430376-c599-4c28-bc72-ffcb86b7da9c\") "
Sep 09 17:38:46.238319 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.238073    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "31430376-c599-4c28-bc72-ffcb86b7da9c" (UID: "31430376-c599-4c28-bc72-ffcb86b7da9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:46.238319 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.238239    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31430376-c599-4c28-bc72-ffcb86b7da9c" (UID: "31430376-c599-4c28-bc72-ffcb86b7da9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:46.238319 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.238293    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-home" (OuterVolumeSpecName: "home") pod "31430376-c599-4c28-bc72-ffcb86b7da9c" (UID: "31430376-c599-4c28-bc72-ffcb86b7da9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:46.238428 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.238325    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:46.238428 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.238347    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:46.240330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.240296    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-dshm" (OuterVolumeSpecName: "dshm") pod "31430376-c599-4c28-bc72-ffcb86b7da9c" (UID: "31430376-c599-4c28-bc72-ffcb86b7da9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:46.240453 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.240349    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31430376-c599-4c28-bc72-ffcb86b7da9c-kube-api-access-2ggrp" (OuterVolumeSpecName: "kube-api-access-2ggrp") pod "31430376-c599-4c28-bc72-ffcb86b7da9c" (UID: "31430376-c599-4c28-bc72-ffcb86b7da9c"). InnerVolumeSpecName "kube-api-access-2ggrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:38:46.240453 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.240400    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31430376-c599-4c28-bc72-ffcb86b7da9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31430376-c599-4c28-bc72-ffcb86b7da9c" (UID: "31430376-c599-4c28-bc72-ffcb86b7da9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:38:46.339773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.339727    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:46.339773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.339764    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ggrp\" (UniqueName: \"kubernetes.io/projected/31430376-c599-4c28-bc72-ffcb86b7da9c-kube-api-access-2ggrp\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:46.339773 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.339778    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31430376-c599-4c28-bc72-ffcb86b7da9c-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:46.340062 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.339789    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31430376-c599-4c28-bc72-ffcb86b7da9c-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:46.734371 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.734334    2558 generic.go:356] "Generic (PLEG): container finished" podID="31430376-c599-4c28-bc72-ffcb86b7da9c" containerID="fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc" exitCode=0
Sep 09 17:38:46.734547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.734381    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"
Sep 09 17:38:46.734547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.734404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd" event={"ID":"31430376-c599-4c28-bc72-ffcb86b7da9c","Type":"ContainerDied","Data":"fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc"}
Sep 09 17:38:46.734547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.734442    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd" event={"ID":"31430376-c599-4c28-bc72-ffcb86b7da9c","Type":"ContainerDied","Data":"385b9038a5e917f06d9e064220959913ab2ab147293c04e6d4d1590fef7bcaa3"}
Sep 09 17:38:46.734547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.734463    2558 scope.go:117] "RemoveContainer" containerID="fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc"
Sep 09 17:38:46.743562 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.743537    2558 scope.go:117] "RemoveContainer" containerID="fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc"
Sep 09 17:38:46.743855 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:38:46.743835    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc\": container with ID starting with fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc not found: ID does not exist" containerID="fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc"
Sep 09 17:38:46.743907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.743863    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc"} err="failed to get container status \"fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc\": rpc error: code = NotFound desc = could not find container \"fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc\": container with ID starting with fa4aeb87382c4fbd9abf2848122f42b3c5551303bd9d597a9d8da480d12e34cc not found: ID does not exist"
Sep 09 17:38:46.755820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.755791    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"]
Sep 09 17:38:46.757779 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.757757    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ea996c/test-model-ea996c-kserve-5bf5ff567-q52hd"]
Sep 09 17:38:46.816076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:46.816041    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31430376-c599-4c28-bc72-ffcb86b7da9c" path="/var/lib/kubelet/pods/31430376-c599-4c28-bc72-ffcb86b7da9c/volumes"
Sep 09 17:38:47.123236 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.123130    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2" podUID="d0524db3-b85a-4922-aebb-9a736bfe9d17" containerName="main" containerID="cri-o://14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a" gracePeriod=45
Sep 09 17:38:47.363941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.363915    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:38:47.449273 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449238    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-dshm\") pod \"d0524db3-b85a-4922-aebb-9a736bfe9d17\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") "
Sep 09 17:38:47.449488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449299    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-model-cache\") pod \"d0524db3-b85a-4922-aebb-9a736bfe9d17\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") "
Sep 09 17:38:47.449488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449325    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0524db3-b85a-4922-aebb-9a736bfe9d17-tls-certs\") pod \"d0524db3-b85a-4922-aebb-9a736bfe9d17\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") "
Sep 09 17:38:47.449488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449345    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-tmp-dir\") pod \"d0524db3-b85a-4922-aebb-9a736bfe9d17\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") "
Sep 09 17:38:47.449488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449385    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-home\") pod \"d0524db3-b85a-4922-aebb-9a736bfe9d17\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") "
Sep 09 17:38:47.449488 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449427    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbtgt\" (UniqueName: \"kubernetes.io/projected/d0524db3-b85a-4922-aebb-9a736bfe9d17-kube-api-access-xbtgt\") pod \"d0524db3-b85a-4922-aebb-9a736bfe9d17\" (UID: \"d0524db3-b85a-4922-aebb-9a736bfe9d17\") "
Sep 09 17:38:47.449743 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449641    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0524db3-b85a-4922-aebb-9a736bfe9d17" (UID: "d0524db3-b85a-4922-aebb-9a736bfe9d17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:47.449743 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449655    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0524db3-b85a-4922-aebb-9a736bfe9d17" (UID: "d0524db3-b85a-4922-aebb-9a736bfe9d17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:47.449743 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.449682    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-home" (OuterVolumeSpecName: "home") pod "d0524db3-b85a-4922-aebb-9a736bfe9d17" (UID: "d0524db3-b85a-4922-aebb-9a736bfe9d17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:47.451656 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.451629    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0524db3-b85a-4922-aebb-9a736bfe9d17-kube-api-access-xbtgt" (OuterVolumeSpecName: "kube-api-access-xbtgt") pod "d0524db3-b85a-4922-aebb-9a736bfe9d17" (UID: "d0524db3-b85a-4922-aebb-9a736bfe9d17"). InnerVolumeSpecName "kube-api-access-xbtgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:38:47.451775 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.451675    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-dshm" (OuterVolumeSpecName: "dshm") pod "d0524db3-b85a-4922-aebb-9a736bfe9d17" (UID: "d0524db3-b85a-4922-aebb-9a736bfe9d17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:38:47.451775 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.451684    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0524db3-b85a-4922-aebb-9a736bfe9d17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0524db3-b85a-4922-aebb-9a736bfe9d17" (UID: "d0524db3-b85a-4922-aebb-9a736bfe9d17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:38:47.550586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.550536    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:47.550586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.550580    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:47.550586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.550591    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtgt\" (UniqueName: \"kubernetes.io/projected/d0524db3-b85a-4922-aebb-9a736bfe9d17-kube-api-access-xbtgt\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:47.550586 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.550600    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:47.550865 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.550608    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0524db3-b85a-4922-aebb-9a736bfe9d17-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:47.550865 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.550616    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0524db3-b85a-4922-aebb-9a736bfe9d17-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:47.741212 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.741126    2558 generic.go:356] "Generic (PLEG): container finished" podID="d0524db3-b85a-4922-aebb-9a736bfe9d17" containerID="14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a" exitCode=0
Sep 09 17:38:47.741212 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.741172    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"
Sep 09 17:38:47.741212 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.741189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2" event={"ID":"d0524db3-b85a-4922-aebb-9a736bfe9d17","Type":"ContainerDied","Data":"14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a"}
Sep 09 17:38:47.741490 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.741235    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2" event={"ID":"d0524db3-b85a-4922-aebb-9a736bfe9d17","Type":"ContainerDied","Data":"7c855e8cb26c8cf1288e5827dcd4c76ea997b5b2137456ac51abac16362e78af"}
Sep 09 17:38:47.741490 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.741254    2558 scope.go:117] "RemoveContainer" containerID="14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a"
Sep 09 17:38:47.754230 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.754207    2558 scope.go:117] "RemoveContainer" containerID="14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a"
Sep 09 17:38:47.754526 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:38:47.754507    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a\": container with ID starting with 14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a not found: ID does not exist" containerID="14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a"
Sep 09 17:38:47.754605 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.754534    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a"} err="failed to get container status \"14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a\": rpc error: code = NotFound desc = could not find container \"14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a\": container with ID starting with 14da32a807ec1133475fe120b3fb3c2d1812dc84b562fb3c8838861efec3085a not found: ID does not exist"
Sep 09 17:38:47.763031 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.763009    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"]
Sep 09 17:38:47.766861 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:47.766838    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-53da12/test-model-53da12-kserve-555bd8bccc-s5zf2"]
Sep 09 17:38:48.821832 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:48.818156    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0524db3-b85a-4922-aebb-9a736bfe9d17" path="/var/lib/kubelet/pods/d0524db3-b85a-4922-aebb-9a736bfe9d17/volumes"
Sep 09 17:38:49.443798 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.443762    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"]
Sep 09 17:38:49.444110 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444098    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31430376-c599-4c28-bc72-ffcb86b7da9c" containerName="main"
Sep 09 17:38:49.444159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444111    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="31430376-c599-4c28-bc72-ffcb86b7da9c" containerName="main"
Sep 09 17:38:49.444159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444119    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba9414f1-3920-489f-b3b6-c0f37c9f445f" containerName="revoke-keys"
Sep 09 17:38:49.444159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444124    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba9414f1-3920-489f-b3b6-c0f37c9f445f" containerName="revoke-keys"
Sep 09 17:38:49.444159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444137    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0524db3-b85a-4922-aebb-9a736bfe9d17" containerName="main"
Sep 09 17:38:49.444159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444143    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0524db3-b85a-4922-aebb-9a736bfe9d17" containerName="main"
Sep 09 17:38:49.444159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444151    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" containerName="payload-processing"
Sep 09 17:38:49.444159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444157    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" containerName="payload-processing"
Sep 09 17:38:49.444368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444208    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="31430376-c599-4c28-bc72-ffcb86b7da9c" containerName="main"
Sep 09 17:38:49.444368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444215    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0524db3-b85a-4922-aebb-9a736bfe9d17" containerName="main"
Sep 09 17:38:49.444368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444221    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba9414f1-3920-489f-b3b6-c0f37c9f445f" containerName="revoke-keys"
Sep 09 17:38:49.444368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.444227    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7aee36-49c8-4c6c-a7a3-538aa314ed8c" containerName="payload-processing"
Sep 09 17:38:49.448409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.448393    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"
Sep 09 17:38:49.451002 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.450947    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:38:49.455083 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.455060    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"]
Sep 09 17:38:49.466388 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.466358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb604e00-c680-4899-bc01-9f8b4d7c8686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm\" (UID: \"fb604e00-c680-4899-bc01-9f8b4d7c8686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"
Sep 09 17:38:49.567308 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.567275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb604e00-c680-4899-bc01-9f8b4d7c8686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm\" (UID: \"fb604e00-c680-4899-bc01-9f8b4d7c8686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"
Sep 09 17:38:49.567891 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.567869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb604e00-c680-4899-bc01-9f8b4d7c8686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm\" (UID: \"fb604e00-c680-4899-bc01-9f8b4d7c8686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"
Sep 09 17:38:49.760387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:49.760307    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"
Sep 09 17:38:50.029527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:50.029498    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"]
Sep 09 17:38:50.031674 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:38:50.031643    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb604e00_c680_4899_bc01_9f8b4d7c8686.slice/crio-d3431b09ab2fb74a8218d568f39d9e1c0ce18b123910922bce1778c066999a3f WatchSource:0}: Error finding container d3431b09ab2fb74a8218d568f39d9e1c0ce18b123910922bce1778c066999a3f: Status 404 returned error can't find the container with id d3431b09ab2fb74a8218d568f39d9e1c0ce18b123910922bce1778c066999a3f
Sep 09 17:38:50.763035 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:50.763004    2558 generic.go:356] "Generic (PLEG): container finished" podID="fb604e00-c680-4899-bc01-9f8b4d7c8686" containerID="7447f726cbb706de21b64527aa3344fbddadc26c0edc262d1d3f5012fefa8154" exitCode=0
Sep 09 17:38:50.763224 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:50.763089    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm" event={"ID":"fb604e00-c680-4899-bc01-9f8b4d7c8686","Type":"ContainerDied","Data":"7447f726cbb706de21b64527aa3344fbddadc26c0edc262d1d3f5012fefa8154"}
Sep 09 17:38:50.763224 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:50.763161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm" event={"ID":"fb604e00-c680-4899-bc01-9f8b4d7c8686","Type":"ContainerStarted","Data":"d3431b09ab2fb74a8218d568f39d9e1c0ce18b123910922bce1778c066999a3f"}
Sep 09 17:38:51.366738 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.366701    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"]
Sep 09 17:38:51.370608 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.370583    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"
Sep 09 17:38:51.376703 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.376667    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"]
Sep 09 17:38:51.483485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.483447    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3372c-fa1a-4c03-aec2-1ae3ea1ded93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd\" (UID: \"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"
Sep 09 17:38:51.585034 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.584995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3372c-fa1a-4c03-aec2-1ae3ea1ded93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd\" (UID: \"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"
Sep 09 17:38:51.585659 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.585637    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3372c-fa1a-4c03-aec2-1ae3ea1ded93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd\" (UID: \"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"
Sep 09 17:38:51.683634 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.683597    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"
Sep 09 17:38:51.901793 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.901766    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"
Sep 09 17:38:51.956211 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.956185    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"]
Sep 09 17:38:51.957849 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:38:51.957816    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a3372c_fa1a_4c03_aec2_1ae3ea1ded93.slice/crio-bde685588f98787b5d1fa34e9abba8400b7e2a24c466ed72138f25a216bac05d WatchSource:0}: Error finding container bde685588f98787b5d1fa34e9abba8400b7e2a24c466ed72138f25a216bac05d: Status 404 returned error can't find the container with id bde685588f98787b5d1fa34e9abba8400b7e2a24c466ed72138f25a216bac05d
Sep 09 17:38:51.987832 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.987808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb604e00-c680-4899-bc01-9f8b4d7c8686-maas-api-service-ca\") pod \"fb604e00-c680-4899-bc01-9f8b4d7c8686\" (UID: \"fb604e00-c680-4899-bc01-9f8b4d7c8686\") "
Sep 09 17:38:51.988144 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:51.988122    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb604e00-c680-4899-bc01-9f8b4d7c8686-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb604e00-c680-4899-bc01-9f8b4d7c8686" (UID: "fb604e00-c680-4899-bc01-9f8b4d7c8686"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:38:52.088878 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:52.088844    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb604e00-c680-4899-bc01-9f8b4d7c8686-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:52.777917 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:52.777885    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"
Sep 09 17:38:52.778349 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:52.777929    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm" event={"ID":"fb604e00-c680-4899-bc01-9f8b4d7c8686","Type":"ContainerDied","Data":"d3431b09ab2fb74a8218d568f39d9e1c0ce18b123910922bce1778c066999a3f"}
Sep 09 17:38:52.778349 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:52.778007    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3431b09ab2fb74a8218d568f39d9e1c0ce18b123910922bce1778c066999a3f"
Sep 09 17:38:52.779640 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:52.779610    2558 generic.go:356] "Generic (PLEG): container finished" podID="43a3372c-fa1a-4c03-aec2-1ae3ea1ded93" containerID="a5b1bec0ecb0d5725110325268ebbdb6dec31174acd7dbe87ab42803be5bf7d6" exitCode=0
Sep 09 17:38:52.779766 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:52.779682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd" event={"ID":"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93","Type":"ContainerDied","Data":"a5b1bec0ecb0d5725110325268ebbdb6dec31174acd7dbe87ab42803be5bf7d6"}
Sep 09 17:38:52.779766 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:52.779714    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd" event={"ID":"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93","Type":"ContainerStarted","Data":"bde685588f98787b5d1fa34e9abba8400b7e2a24c466ed72138f25a216bac05d"}
Sep 09 17:38:53.909993 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:53.909966    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"
Sep 09 17:38:54.006720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:54.006682    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3372c-fa1a-4c03-aec2-1ae3ea1ded93-maas-api-service-ca\") pod \"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93\" (UID: \"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93\") "
Sep 09 17:38:54.007150 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:54.007114    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a3372c-fa1a-4c03-aec2-1ae3ea1ded93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43a3372c-fa1a-4c03-aec2-1ae3ea1ded93" (UID: "43a3372c-fa1a-4c03-aec2-1ae3ea1ded93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:38:54.108399 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:54.108322    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3372c-fa1a-4c03-aec2-1ae3ea1ded93-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:38:54.790671 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:54.790643    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"
Sep 09 17:38:54.790840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:54.790670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd" event={"ID":"43a3372c-fa1a-4c03-aec2-1ae3ea1ded93","Type":"ContainerDied","Data":"bde685588f98787b5d1fa34e9abba8400b7e2a24c466ed72138f25a216bac05d"}
Sep 09 17:38:54.790840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:54.790712    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bde685588f98787b5d1fa34e9abba8400b7e2a24c466ed72138f25a216bac05d"
Sep 09 17:38:56.988585 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.988549    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"]
Sep 09 17:38:56.989052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.988901    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb604e00-c680-4899-bc01-9f8b4d7c8686" containerName="revoke-keys"
Sep 09 17:38:56.989052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.988915    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb604e00-c680-4899-bc01-9f8b4d7c8686" containerName="revoke-keys"
Sep 09 17:38:56.989052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.988928    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a3372c-fa1a-4c03-aec2-1ae3ea1ded93" containerName="revoke-keys"
Sep 09 17:38:56.989052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.988935    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a3372c-fa1a-4c03-aec2-1ae3ea1ded93" containerName="revoke-keys"
Sep 09 17:38:56.989052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.989040    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb604e00-c680-4899-bc01-9f8b4d7c8686" containerName="revoke-keys"
Sep 09 17:38:56.989052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.989052    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="43a3372c-fa1a-4c03-aec2-1ae3ea1ded93" containerName="revoke-keys"
Sep 09 17:38:56.992133 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.992118    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:56.994535 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.994513    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-12bac2\"/\"noten-model-12bac2-kserve-self-signed-certs\""
Sep 09 17:38:56.995778 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.995760    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-12bac2\"/\"kube-root-ca.crt\""
Sep 09 17:38:56.995967 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.995919    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-12bac2\"/\"default-dockercfg-88mxp\""
Sep 09 17:38:56.996077 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.996059    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-12bac2\"/\"openshift-service-ca.crt\""
Sep 09 17:38:56.999713 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:56.999691    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"]
Sep 09 17:38:57.033924 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.033886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-tmp-dir\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.034097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.033927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-model-cache\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.034097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.034017    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce6ba20-7fde-45f4-8252-d76b504486d3-tls-certs\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.034097 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.034082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-dshm\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.034233 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.034101    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-home\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.034233 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.034133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcs7x\" (UniqueName: \"kubernetes.io/projected/bce6ba20-7fde-45f4-8252-d76b504486d3-kube-api-access-fcs7x\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135138 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-tmp-dir\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135138 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-model-cache\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce6ba20-7fde-45f4-8252-d76b504486d3-tls-certs\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-dshm\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-home\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcs7x\" (UniqueName: \"kubernetes.io/projected/bce6ba20-7fde-45f4-8252-d76b504486d3-kube-api-access-fcs7x\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135623 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-tmp-dir\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135623 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135575    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-model-cache\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.135715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.135680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-home\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.137479 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.137455    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-dshm\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.137684 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.137666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce6ba20-7fde-45f4-8252-d76b504486d3-tls-certs\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.142898 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.142878    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcs7x\" (UniqueName: \"kubernetes.io/projected/bce6ba20-7fde-45f4-8252-d76b504486d3-kube-api-access-fcs7x\") pod \"noten-model-12bac2-kserve-59995d976c-2qjxc\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") " pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.301883 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.301783    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:38:57.574648 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.574620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"]
Sep 09 17:38:57.576784 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:38:57.576757    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbce6ba20_7fde_45f4_8252_d76b504486d3.slice/crio-44b291417c5e27e507e3c480c252e1f0dea6defd7dc94abab995f76cd8c86bf1 WatchSource:0}: Error finding container 44b291417c5e27e507e3c480c252e1f0dea6defd7dc94abab995f76cd8c86bf1: Status 404 returned error can't find the container with id 44b291417c5e27e507e3c480c252e1f0dea6defd7dc94abab995f76cd8c86bf1
Sep 09 17:38:57.809372 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.809337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc" event={"ID":"bce6ba20-7fde-45f4-8252-d76b504486d3","Type":"ContainerStarted","Data":"96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039"}
Sep 09 17:38:57.809372 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.809377    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc" event={"ID":"bce6ba20-7fde-45f4-8252-d76b504486d3","Type":"ContainerStarted","Data":"44b291417c5e27e507e3c480c252e1f0dea6defd7dc94abab995f76cd8c86bf1"}
Sep 09 17:38:57.827156 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:38:57.827015    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc" podStartSLOduration=1.8269988499999998 podStartE2EDuration="1.82699885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:38:57.824603778 +0000 UTC m=+1313.452756579" watchObservedRunningTime="2026-09-09 17:38:57.82699885 +0000 UTC m=+1313.455151552"
Sep 09 17:39:00.132024 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.131980    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"]
Sep 09 17:39:00.132473 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.132231    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw" podUID="c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" containerName="payload-pre-processing" containerID="cri-o://002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e" gracePeriod=30
Sep 09 17:39:00.280171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.280135    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"]
Sep 09 17:39:00.282100 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.282070    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-53da12-kzkkm"]
Sep 09 17:39:00.379998 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.379971    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:39:00.463561 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.463531    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-plugins-config-volume\") pod \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") "
Sep 09 17:39:00.463750 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.463573    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwqh5\" (UniqueName: \"kubernetes.io/projected/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-kube-api-access-hwqh5\") pod \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\" (UID: \"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7\") "
Sep 09 17:39:00.463988 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.463936    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" (UID: "c892746e-47d6-4c03-b0e1-47e2b1f8d0e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:39:00.465593 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.465572    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-kube-api-access-hwqh5" (OuterVolumeSpecName: "kube-api-access-hwqh5") pod "c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" (UID: "c892746e-47d6-4c03-b0e1-47e2b1f8d0e7"). InnerVolumeSpecName "kube-api-access-hwqh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:39:00.564545 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.564509    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqh5\" (UniqueName: \"kubernetes.io/projected/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-kube-api-access-hwqh5\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:00.564545 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.564536    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:00.816888 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.816805    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb604e00-c680-4899-bc01-9f8b4d7c8686" path="/var/lib/kubelet/pods/fb604e00-c680-4899-bc01-9f8b4d7c8686/volumes"
Sep 09 17:39:00.827197 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.827170    2558 generic.go:356] "Generic (PLEG): container finished" podID="c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" containerID="002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e" exitCode=0
Sep 09 17:39:00.827307 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.827212    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"
Sep 09 17:39:00.827307 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.827253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw" event={"ID":"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7","Type":"ContainerDied","Data":"002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e"}
Sep 09 17:39:00.827383 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.827305    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw" event={"ID":"c892746e-47d6-4c03-b0e1-47e2b1f8d0e7","Type":"ContainerDied","Data":"ed172ab8fcef359692ab7a42daf4debcf18e4be0dd441c769994fb1268f4fa43"}
Sep 09 17:39:00.827383 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.827329    2558 scope.go:117] "RemoveContainer" containerID="002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e"
Sep 09 17:39:00.836391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.836374    2558 scope.go:117] "RemoveContainer" containerID="002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e"
Sep 09 17:39:00.836677 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:39:00.836657    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e\": container with ID starting with 002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e not found: ID does not exist" containerID="002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e"
Sep 09 17:39:00.836712 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.836687    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e"} err="failed to get container status \"002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e\": rpc error: code = NotFound desc = could not find container \"002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e\": container with ID starting with 002f38e4f8540c139ddab3d735b3cdd1279c74b5eb79b59b46aca4fbcc5c4e5e not found: ID does not exist"
Sep 09 17:39:00.849077 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.849056    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"]
Sep 09 17:39:00.852844 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:00.852810    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-53da12-79d4dc75f4-pmjfw"]
Sep 09 17:39:02.254004 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.253968    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"]
Sep 09 17:39:02.254407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.254223    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5" podUID="3d8ce7ed-5d44-420a-aec5-132e004f7bb3" containerName="payload-processing" containerID="cri-o://f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f" gracePeriod=30
Sep 09 17:39:02.258526 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.258501    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"]
Sep 09 17:39:02.258884 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.258856    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx" podUID="78131cdf-9a89-41a1-bb75-33833866737b" containerName="payload-pre-processing" containerID="cri-o://db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37" gracePeriod=30
Sep 09 17:39:02.417291 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.417247    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"]
Sep 09 17:39:02.420593 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.420562    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71f58f92-mbvwd"]
Sep 09 17:39:02.516519 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.516449    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:39:02.545438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.545414    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:39:02.582897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.582868    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfslk\" (UniqueName: \"kubernetes.io/projected/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-kube-api-access-qfslk\") pod \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") "
Sep 09 17:39:02.583052 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.582907    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-plugins-config-volume\") pod \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\" (UID: \"3d8ce7ed-5d44-420a-aec5-132e004f7bb3\") "
Sep 09 17:39:02.583328 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.583302    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d8ce7ed-5d44-420a-aec5-132e004f7bb3" (UID: "3d8ce7ed-5d44-420a-aec5-132e004f7bb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:39:02.584902 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.584883    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-kube-api-access-qfslk" (OuterVolumeSpecName: "kube-api-access-qfslk") pod "3d8ce7ed-5d44-420a-aec5-132e004f7bb3" (UID: "3d8ce7ed-5d44-420a-aec5-132e004f7bb3"). InnerVolumeSpecName "kube-api-access-qfslk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:39:02.683525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.683493    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78131cdf-9a89-41a1-bb75-33833866737b-plugins-config-volume\") pod \"78131cdf-9a89-41a1-bb75-33833866737b\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") "
Sep 09 17:39:02.683681 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.683535    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2l76\" (UniqueName: \"kubernetes.io/projected/78131cdf-9a89-41a1-bb75-33833866737b-kube-api-access-p2l76\") pod \"78131cdf-9a89-41a1-bb75-33833866737b\" (UID: \"78131cdf-9a89-41a1-bb75-33833866737b\") "
Sep 09 17:39:02.683745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.683727    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfslk\" (UniqueName: \"kubernetes.io/projected/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-kube-api-access-qfslk\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:02.683784 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.683752    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8ce7ed-5d44-420a-aec5-132e004f7bb3-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:02.683897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.683863    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78131cdf-9a89-41a1-bb75-33833866737b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78131cdf-9a89-41a1-bb75-33833866737b" (UID: "78131cdf-9a89-41a1-bb75-33833866737b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:39:02.685736 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.685716    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78131cdf-9a89-41a1-bb75-33833866737b-kube-api-access-p2l76" (OuterVolumeSpecName: "kube-api-access-p2l76") pod "78131cdf-9a89-41a1-bb75-33833866737b" (UID: "78131cdf-9a89-41a1-bb75-33833866737b"). InnerVolumeSpecName "kube-api-access-p2l76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:39:02.784493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.784412    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78131cdf-9a89-41a1-bb75-33833866737b-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:02.784493 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.784436    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2l76\" (UniqueName: \"kubernetes.io/projected/78131cdf-9a89-41a1-bb75-33833866737b-kube-api-access-p2l76\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:02.815738 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.815706    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43a3372c-fa1a-4c03-aec2-1ae3ea1ded93" path="/var/lib/kubelet/pods/43a3372c-fa1a-4c03-aec2-1ae3ea1ded93/volumes"
Sep 09 17:39:02.816080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.816067    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" path="/var/lib/kubelet/pods/c892746e-47d6-4c03-b0e1-47e2b1f8d0e7/volumes"
Sep 09 17:39:02.840983 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.840936    2558 generic.go:356] "Generic (PLEG): container finished" podID="78131cdf-9a89-41a1-bb75-33833866737b" containerID="db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37" exitCode=0
Sep 09 17:39:02.841116 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.841007    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"
Sep 09 17:39:02.841116 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.841020    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx" event={"ID":"78131cdf-9a89-41a1-bb75-33833866737b","Type":"ContainerDied","Data":"db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37"}
Sep 09 17:39:02.841116 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.841075    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx" event={"ID":"78131cdf-9a89-41a1-bb75-33833866737b","Type":"ContainerDied","Data":"ba64eea79d5646fa6ec364659776ab156186040f44ed295a91e2a5cb9a2b7ede"}
Sep 09 17:39:02.841116 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.841097    2558 scope.go:117] "RemoveContainer" containerID="db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37"
Sep 09 17:39:02.842845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.842818    2558 generic.go:356] "Generic (PLEG): container finished" podID="3d8ce7ed-5d44-420a-aec5-132e004f7bb3" containerID="f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f" exitCode=0
Sep 09 17:39:02.842969 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.842859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5" event={"ID":"3d8ce7ed-5d44-420a-aec5-132e004f7bb3","Type":"ContainerDied","Data":"f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f"}
Sep 09 17:39:02.842969 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.842899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5" event={"ID":"3d8ce7ed-5d44-420a-aec5-132e004f7bb3","Type":"ContainerDied","Data":"b0c813166e4649a6ca8b64ca25a63dd06b0f34f56bb4fb963574da3d9125c654"}
Sep 09 17:39:02.843111 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.842979    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"
Sep 09 17:39:02.850908 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.850892    2558 scope.go:117] "RemoveContainer" containerID="db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37"
Sep 09 17:39:02.851210 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:39:02.851193    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37\": container with ID starting with db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37 not found: ID does not exist" containerID="db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37"
Sep 09 17:39:02.851256 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.851221    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37"} err="failed to get container status \"db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37\": rpc error: code = NotFound desc = could not find container \"db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37\": container with ID starting with db149eecb86bcebc888f20198810c5108f0ce03174abad15952976e8b7584f37 not found: ID does not exist"
Sep 09 17:39:02.851256 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.851238    2558 scope.go:117] "RemoveContainer" containerID="f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f"
Sep 09 17:39:02.859339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.859321    2558 scope.go:117] "RemoveContainer" containerID="f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f"
Sep 09 17:39:02.859610 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:39:02.859591    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f\": container with ID starting with f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f not found: ID does not exist" containerID="f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f"
Sep 09 17:39:02.859667 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.859619    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f"} err="failed to get container status \"f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f\": rpc error: code = NotFound desc = could not find container \"f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f\": container with ID starting with f4b8faddcb063036fcbbd0f4deb1b3ae8a0a7876fb8470036fda5d944873b30f not found: ID does not exist"
Sep 09 17:39:02.862433 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.862412    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"]
Sep 09 17:39:02.866575 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.866556    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71f58f92-7496f958b-d7fpx"]
Sep 09 17:39:02.876144 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.876125    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"]
Sep 09 17:39:02.880483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:02.880460    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71f58f92-dfd6f4f45-6wbd5"]
Sep 09 17:39:04.816109 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:04.816081    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8ce7ed-5d44-420a-aec5-132e004f7bb3" path="/var/lib/kubelet/pods/3d8ce7ed-5d44-420a-aec5-132e004f7bb3/volumes"
Sep 09 17:39:04.816474 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:04.816453    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78131cdf-9a89-41a1-bb75-33833866737b" path="/var/lib/kubelet/pods/78131cdf-9a89-41a1-bb75-33833866737b/volumes"
Sep 09 17:39:06.062228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:06.062149    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"]
Sep 09 17:39:07.302402 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:07.302352    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:39:12.349936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.349898    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"]
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350210    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" containerName="payload-pre-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350221    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" containerName="payload-pre-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350232    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78131cdf-9a89-41a1-bb75-33833866737b" containerName="payload-pre-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350238    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="78131cdf-9a89-41a1-bb75-33833866737b" containerName="payload-pre-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350250    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8ce7ed-5d44-420a-aec5-132e004f7bb3" containerName="payload-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350255    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8ce7ed-5d44-420a-aec5-132e004f7bb3" containerName="payload-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350305    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="78131cdf-9a89-41a1-bb75-33833866737b" containerName="payload-pre-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350315    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8ce7ed-5d44-420a-aec5-132e004f7bb3" containerName="payload-processing"
Sep 09 17:39:12.350339 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.350323    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c892746e-47d6-4c03-b0e1-47e2b1f8d0e7" containerName="payload-pre-processing"
Sep 09 17:39:12.354813 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.354793    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.357812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.357793    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-480517\""
Sep 09 17:39:12.357973 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.357853    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-480517-dockercfg-svbmt\""
Sep 09 17:39:12.363936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.363907    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"]
Sep 09 17:39:12.387753 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.387719    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"]
Sep 09 17:39:12.390920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.390904    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.401456 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.401424    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"]
Sep 09 17:39:12.465241 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.465209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9016e1-79c5-48be-a043-4102fc23119c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.465241 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.465243    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xgft\" (UniqueName: \"kubernetes.io/projected/9e9016e1-79c5-48be-a043-4102fc23119c-kube-api-access-7xgft\") pod \"payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.465506 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.465268    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf12f5e8-08b6-4de5-9244-9c3b3677271b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.465506 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.465427    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhxj\" (UniqueName: \"kubernetes.io/projected/bf12f5e8-08b6-4de5-9244-9c3b3677271b-kube-api-access-bmhxj\") pod \"payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.565929 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.565884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9016e1-79c5-48be-a043-4102fc23119c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.565929 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.565925    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgft\" (UniqueName: \"kubernetes.io/projected/9e9016e1-79c5-48be-a043-4102fc23119c-kube-api-access-7xgft\") pod \"payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.566213 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.565946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf12f5e8-08b6-4de5-9244-9c3b3677271b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.566213 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.566028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhxj\" (UniqueName: \"kubernetes.io/projected/bf12f5e8-08b6-4de5-9244-9c3b3677271b-kube-api-access-bmhxj\") pod \"payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.566682 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.566658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9016e1-79c5-48be-a043-4102fc23119c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.566682 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.566673    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf12f5e8-08b6-4de5-9244-9c3b3677271b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.574481 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.574445    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgft\" (UniqueName: \"kubernetes.io/projected/9e9016e1-79c5-48be-a043-4102fc23119c-kube-api-access-7xgft\") pod \"payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.574594 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.574505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhxj\" (UniqueName: \"kubernetes.io/projected/bf12f5e8-08b6-4de5-9244-9c3b3677271b-kube-api-access-bmhxj\") pod \"payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.666098 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.666009    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:12.701232 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.701194    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:12.956627 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:12.956602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"]
Sep 09 17:39:12.958316 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:39:12.958283    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9016e1_79c5_48be_a043_4102fc23119c.slice/crio-4ca9860d3d3e6ecf2abbc6d269e8d1c38684f03ef636f1f745ff50ae7d752fc3 WatchSource:0}: Error finding container 4ca9860d3d3e6ecf2abbc6d269e8d1c38684f03ef636f1f745ff50ae7d752fc3: Status 404 returned error can't find the container with id 4ca9860d3d3e6ecf2abbc6d269e8d1c38684f03ef636f1f745ff50ae7d752fc3
Sep 09 17:39:13.013000 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.012972    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"]
Sep 09 17:39:13.013990 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:39:13.013935    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf12f5e8_08b6_4de5_9244_9c3b3677271b.slice/crio-44f70d5c4bf8f9e97efaa3f8f088313ae8c5a3a289452513c0f589c51c3155e7 WatchSource:0}: Error finding container 44f70d5c4bf8f9e97efaa3f8f088313ae8c5a3a289452513c0f589c51c3155e7: Status 404 returned error can't find the container with id 44f70d5c4bf8f9e97efaa3f8f088313ae8c5a3a289452513c0f589c51c3155e7
Sep 09 17:39:13.911159 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.911111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg" event={"ID":"9e9016e1-79c5-48be-a043-4102fc23119c","Type":"ContainerStarted","Data":"f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe"}
Sep 09 17:39:13.911713 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.911184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg" event={"ID":"9e9016e1-79c5-48be-a043-4102fc23119c","Type":"ContainerStarted","Data":"4ca9860d3d3e6ecf2abbc6d269e8d1c38684f03ef636f1f745ff50ae7d752fc3"}
Sep 09 17:39:13.911713 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.911214    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:13.912812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.912792    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67" event={"ID":"bf12f5e8-08b6-4de5-9244-9c3b3677271b","Type":"ContainerStarted","Data":"bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6"}
Sep 09 17:39:13.912920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.912827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67" event={"ID":"bf12f5e8-08b6-4de5-9244-9c3b3677271b","Type":"ContainerStarted","Data":"44f70d5c4bf8f9e97efaa3f8f088313ae8c5a3a289452513c0f589c51c3155e7"}
Sep 09 17:39:13.912920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.912875    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:13.931698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.931651    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg" podStartSLOduration=1.931635951 podStartE2EDuration="1.931635951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:13.930748291 +0000 UTC m=+1329.558900995" watchObservedRunningTime="2026-09-09 17:39:13.931635951 +0000 UTC m=+1329.559788698"
Sep 09 17:39:13.949398 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:13.949353    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67" podStartSLOduration=1.9493386 podStartE2EDuration="1.9493386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:13.947374743 +0000 UTC m=+1329.575527458" watchObservedRunningTime="2026-09-09 17:39:13.9493386 +0000 UTC m=+1329.577491302"
Sep 09 17:39:21.082229 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.082159    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc" podUID="bce6ba20-7fde-45f4-8252-d76b504486d3" containerName="main" containerID="cri-o://96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039" gracePeriod=45
Sep 09 17:39:21.326040 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.326009    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:39:21.451189 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451145    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce6ba20-7fde-45f4-8252-d76b504486d3-tls-certs\") pod \"bce6ba20-7fde-45f4-8252-d76b504486d3\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") "
Sep 09 17:39:21.451438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451220    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-model-cache\") pod \"bce6ba20-7fde-45f4-8252-d76b504486d3\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") "
Sep 09 17:39:21.451438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451253    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-home\") pod \"bce6ba20-7fde-45f4-8252-d76b504486d3\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") "
Sep 09 17:39:21.451438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-tmp-dir\") pod \"bce6ba20-7fde-45f4-8252-d76b504486d3\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") "
Sep 09 17:39:21.451438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451306    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-dshm\") pod \"bce6ba20-7fde-45f4-8252-d76b504486d3\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") "
Sep 09 17:39:21.451438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451335    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcs7x\" (UniqueName: \"kubernetes.io/projected/bce6ba20-7fde-45f4-8252-d76b504486d3-kube-api-access-fcs7x\") pod \"bce6ba20-7fde-45f4-8252-d76b504486d3\" (UID: \"bce6ba20-7fde-45f4-8252-d76b504486d3\") "
Sep 09 17:39:21.451695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451522    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "bce6ba20-7fde-45f4-8252-d76b504486d3" (UID: "bce6ba20-7fde-45f4-8252-d76b504486d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:39:21.451695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451530    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-home" (OuterVolumeSpecName: "home") pod "bce6ba20-7fde-45f4-8252-d76b504486d3" (UID: "bce6ba20-7fde-45f4-8252-d76b504486d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:39:21.451695 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.451557    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bce6ba20-7fde-45f4-8252-d76b504486d3" (UID: "bce6ba20-7fde-45f4-8252-d76b504486d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:39:21.453642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.453577    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-dshm" (OuterVolumeSpecName: "dshm") pod "bce6ba20-7fde-45f4-8252-d76b504486d3" (UID: "bce6ba20-7fde-45f4-8252-d76b504486d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:39:21.453642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.453593    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce6ba20-7fde-45f4-8252-d76b504486d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bce6ba20-7fde-45f4-8252-d76b504486d3" (UID: "bce6ba20-7fde-45f4-8252-d76b504486d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:39:21.453825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.453675    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce6ba20-7fde-45f4-8252-d76b504486d3-kube-api-access-fcs7x" (OuterVolumeSpecName: "kube-api-access-fcs7x") pod "bce6ba20-7fde-45f4-8252-d76b504486d3" (UID: "bce6ba20-7fde-45f4-8252-d76b504486d3"). InnerVolumeSpecName "kube-api-access-fcs7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:39:21.552866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.552813    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcs7x\" (UniqueName: \"kubernetes.io/projected/bce6ba20-7fde-45f4-8252-d76b504486d3-kube-api-access-fcs7x\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:21.552866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.552860    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce6ba20-7fde-45f4-8252-d76b504486d3-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:21.552866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.552872    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:21.552866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.552882    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:21.552866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.552889    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:21.553200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.552896    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce6ba20-7fde-45f4-8252-d76b504486d3-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:39:21.956555 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.956518    2558 generic.go:356] "Generic (PLEG): container finished" podID="bce6ba20-7fde-45f4-8252-d76b504486d3" containerID="96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039" exitCode=0
Sep 09 17:39:21.956736 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.956567    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"
Sep 09 17:39:21.956736 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.956609    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc" event={"ID":"bce6ba20-7fde-45f4-8252-d76b504486d3","Type":"ContainerDied","Data":"96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039"}
Sep 09 17:39:21.956736 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.956656    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc" event={"ID":"bce6ba20-7fde-45f4-8252-d76b504486d3","Type":"ContainerDied","Data":"44b291417c5e27e507e3c480c252e1f0dea6defd7dc94abab995f76cd8c86bf1"}
Sep 09 17:39:21.956736 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.956673    2558 scope.go:117] "RemoveContainer" containerID="96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039"
Sep 09 17:39:21.966059 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.966041    2558 scope.go:117] "RemoveContainer" containerID="96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039"
Sep 09 17:39:21.966319 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:39:21.966298    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039\": container with ID starting with 96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039 not found: ID does not exist" containerID="96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039"
Sep 09 17:39:21.966372 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.966328    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039"} err="failed to get container status \"96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039\": rpc error: code = NotFound desc = could not find container \"96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039\": container with ID starting with 96103bf57b5485fbc25f92177fced245350357d14ebb109966c2a91c4253d039 not found: ID does not exist"
Sep 09 17:39:21.982528 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.982501    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"]
Sep 09 17:39:21.985492 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:21.985469    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-12bac2/noten-model-12bac2-kserve-59995d976c-2qjxc"]
Sep 09 17:39:22.816694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:22.816658    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce6ba20-7fde-45f4-8252-d76b504486d3" path="/var/lib/kubelet/pods/bce6ba20-7fde-45f4-8252-d76b504486d3/volumes"
Sep 09 17:39:24.921694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:24.921660    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:39:24.922085 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:24.921736    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:39:27.554741 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.554654    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"]
Sep 09 17:39:27.555278 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.555205    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce6ba20-7fde-45f4-8252-d76b504486d3" containerName="main"
Sep 09 17:39:27.555278 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.555225    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce6ba20-7fde-45f4-8252-d76b504486d3" containerName="main"
Sep 09 17:39:27.555393 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.555329    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce6ba20-7fde-45f4-8252-d76b504486d3" containerName="main"
Sep 09 17:39:27.559933 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.559911    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.568099 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.568076    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6b44e3-dockercfg-ffkxc\""
Sep 09 17:39:27.568216 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.568180    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6b44e3\""
Sep 09 17:39:27.572911 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.572884    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"]
Sep 09 17:39:27.592178 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.592142    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"]
Sep 09 17:39:27.595320 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.595302    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:27.603239 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.603209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:27.603385 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.603288    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsgp\" (UniqueName: \"kubernetes.io/projected/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-kube-api-access-flsgp\") pod \"payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:27.603385 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.603336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ctk\" (UniqueName: \"kubernetes.io/projected/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-kube-api-access-45ctk\") pod \"payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.603501 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.603412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.605082 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.605057    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"]
Sep 09 17:39:27.703830 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.703793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.703830 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.703837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:27.704115 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.703873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsgp\" (UniqueName: \"kubernetes.io/projected/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-kube-api-access-flsgp\") pod \"payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:27.704115 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.703901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ctk\" (UniqueName: \"kubernetes.io/projected/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-kube-api-access-45ctk\") pod \"payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.704615 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.704597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:27.704656 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.704613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.712526 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.712491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsgp\" (UniqueName: \"kubernetes.io/projected/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-kube-api-access-flsgp\") pod \"payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:27.712711 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.712687    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ctk\" (UniqueName: \"kubernetes.io/projected/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-kube-api-access-45ctk\") pod \"payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.870280 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.870191    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:27.906579 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:27.906546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:28.147223 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:28.147190    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"]
Sep 09 17:39:28.148544 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:39:28.148513    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59df4e61_4e6d_4452_b9d8_f4dfbdcfb3ae.slice/crio-11c70169f080289cc0e38d231c69c5ec8c028efb0f815feafebc71261163a172 WatchSource:0}: Error finding container 11c70169f080289cc0e38d231c69c5ec8c028efb0f815feafebc71261163a172: Status 404 returned error can't find the container with id 11c70169f080289cc0e38d231c69c5ec8c028efb0f815feafebc71261163a172
Sep 09 17:39:28.199734 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:28.199709    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"]
Sep 09 17:39:28.202171 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:39:28.202141    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37f9c4b_9fc1_49eb_b8cc_cff52287ba1b.slice/crio-629c47fee9fcd1a75ae85dd096afb283926e94def08c143a9ceba453af274ee8 WatchSource:0}: Error finding container 629c47fee9fcd1a75ae85dd096afb283926e94def08c143a9ceba453af274ee8: Status 404 returned error can't find the container with id 629c47fee9fcd1a75ae85dd096afb283926e94def08c143a9ceba453af274ee8
Sep 09 17:39:29.003615 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.003568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg" event={"ID":"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae","Type":"ContainerStarted","Data":"4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb"}
Sep 09 17:39:29.004119 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.003766    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:29.004119 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.003805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg" event={"ID":"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae","Type":"ContainerStarted","Data":"11c70169f080289cc0e38d231c69c5ec8c028efb0f815feafebc71261163a172"}
Sep 09 17:39:29.005296 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.005272    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt" event={"ID":"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b","Type":"ContainerStarted","Data":"1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90"}
Sep 09 17:39:29.005359 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.005317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt" event={"ID":"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b","Type":"ContainerStarted","Data":"629c47fee9fcd1a75ae85dd096afb283926e94def08c143a9ceba453af274ee8"}
Sep 09 17:39:29.005359 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.005345    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:29.020497 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.020449    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg" podStartSLOduration=2.020435237 podStartE2EDuration="2.020435237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:29.01838244 +0000 UTC m=+1344.646535158" watchObservedRunningTime="2026-09-09 17:39:29.020435237 +0000 UTC m=+1344.648587939"
Sep 09 17:39:29.034776 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:29.034730    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt" podStartSLOduration=2.034715735 podStartE2EDuration="2.034715735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:29.03341923 +0000 UTC m=+1344.661571931" watchObservedRunningTime="2026-09-09 17:39:29.034715735 +0000 UTC m=+1344.662868438"
Sep 09 17:39:40.012615 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.012583    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:39:40.013045 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.012661    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:39:40.386199 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.386112    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"]
Sep 09 17:39:40.389263 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.389248    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.393602 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.393581    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-17e67b\""
Sep 09 17:39:40.393731 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.393684    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-17e67b-dockercfg-tbb6k\""
Sep 09 17:39:40.401579 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.401557    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"]
Sep 09 17:39:40.427300 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.427272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pqv\" (UniqueName: \"kubernetes.io/projected/07e33598-3b60-42b9-bfc2-c282300a8d7d-kube-api-access-48pqv\") pod \"payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.427443 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.427326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e33598-3b60-42b9-bfc2-c282300a8d7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.527781 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.527748    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pqv\" (UniqueName: \"kubernetes.io/projected/07e33598-3b60-42b9-bfc2-c282300a8d7d-kube-api-access-48pqv\") pod \"payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.527973 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.527799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e33598-3b60-42b9-bfc2-c282300a8d7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.528618 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.528597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e33598-3b60-42b9-bfc2-c282300a8d7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.535444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.535419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pqv\" (UniqueName: \"kubernetes.io/projected/07e33598-3b60-42b9-bfc2-c282300a8d7d-kube-api-access-48pqv\") pod \"payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.699835 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.699805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:40.972125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:40.972098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"]
Sep 09 17:39:40.974436 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:39:40.974407    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e33598_3b60_42b9_bfc2_c282300a8d7d.slice/crio-2c2ed4cb26ac31a8510ca523b66ff0fd035df0ff131f91397ce50c77faf44a34 WatchSource:0}: Error finding container 2c2ed4cb26ac31a8510ca523b66ff0fd035df0ff131f91397ce50c77faf44a34: Status 404 returned error can't find the container with id 2c2ed4cb26ac31a8510ca523b66ff0fd035df0ff131f91397ce50c77faf44a34
Sep 09 17:39:41.074551 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:41.074515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8" event={"ID":"07e33598-3b60-42b9-bfc2-c282300a8d7d","Type":"ContainerStarted","Data":"aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856"}
Sep 09 17:39:41.074907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:41.074571    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8" event={"ID":"07e33598-3b60-42b9-bfc2-c282300a8d7d","Type":"ContainerStarted","Data":"2c2ed4cb26ac31a8510ca523b66ff0fd035df0ff131f91397ce50c77faf44a34"}
Sep 09 17:39:41.074907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:41.074623    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:41.090166 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:41.090115    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8" podStartSLOduration=1.090098398 podStartE2EDuration="1.090098398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:41.087561727 +0000 UTC m=+1356.715714443" watchObservedRunningTime="2026-09-09 17:39:41.090098398 +0000 UTC m=+1356.718251102"
Sep 09 17:39:52.082156 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:52.082126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:39:55.615333 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.615299    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"]
Sep 09 17:39:55.619820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.619804    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.622243 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.622222    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-607751\""
Sep 09 17:39:55.629694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.629630    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"]
Sep 09 17:39:55.664549 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.664514    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"]
Sep 09 17:39:55.667939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.667923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:55.670405 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.670383    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-607751-dockercfg-r7c62\""
Sep 09 17:39:55.677523 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.677503    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"]
Sep 09 17:39:55.758821 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.758790    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6299fa-a3cb-49e4-84d2-448abadf0784-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.758821 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.758821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx67v\" (UniqueName: \"kubernetes.io/projected/10e69cbb-2904-4ce0-a62b-ccac89a78922-kube-api-access-lx67v\") pod \"payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:55.759027 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.758853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e69cbb-2904-4ce0-a62b-ccac89a78922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:55.759027 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.758940    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkz5w\" (UniqueName: \"kubernetes.io/projected/4f6299fa-a3cb-49e4-84d2-448abadf0784-kube-api-access-pkz5w\") pod \"payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.859941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.859901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6299fa-a3cb-49e4-84d2-448abadf0784-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.859941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.859943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx67v\" (UniqueName: \"kubernetes.io/projected/10e69cbb-2904-4ce0-a62b-ccac89a78922-kube-api-access-lx67v\") pod \"payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:55.860217 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.859996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e69cbb-2904-4ce0-a62b-ccac89a78922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:55.860217 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.860043    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkz5w\" (UniqueName: \"kubernetes.io/projected/4f6299fa-a3cb-49e4-84d2-448abadf0784-kube-api-access-pkz5w\") pod \"payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.860736 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.860714    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6299fa-a3cb-49e4-84d2-448abadf0784-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.860794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.860720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e69cbb-2904-4ce0-a62b-ccac89a78922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:55.867609 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.867548    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx67v\" (UniqueName: \"kubernetes.io/projected/10e69cbb-2904-4ce0-a62b-ccac89a78922-kube-api-access-lx67v\") pod \"payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:55.867742 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.867718    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkz5w\" (UniqueName: \"kubernetes.io/projected/4f6299fa-a3cb-49e4-84d2-448abadf0784-kube-api-access-pkz5w\") pod \"payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.931584 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.931548    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:55.977973 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:55.977919    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:56.203913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:56.203883    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"]
Sep 09 17:39:56.205800 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:39:56.205773    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6299fa_a3cb_49e4_84d2_448abadf0784.slice/crio-a2d1a8505685e3dbb2f4bf905fe0e9bfc90983be70ad103677c5c741e56bfc14 WatchSource:0}: Error finding container a2d1a8505685e3dbb2f4bf905fe0e9bfc90983be70ad103677c5c741e56bfc14: Status 404 returned error can't find the container with id a2d1a8505685e3dbb2f4bf905fe0e9bfc90983be70ad103677c5c741e56bfc14
Sep 09 17:39:56.268276 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:56.268241    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"]
Sep 09 17:39:56.269572 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:39:56.269532    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e69cbb_2904_4ce0_a62b_ccac89a78922.slice/crio-ec8fb963ee28ac114cfa4d947b1b1c26da52d42ec66d66d00d95ae8d4c9b4f78 WatchSource:0}: Error finding container ec8fb963ee28ac114cfa4d947b1b1c26da52d42ec66d66d00d95ae8d4c9b4f78: Status 404 returned error can't find the container with id ec8fb963ee28ac114cfa4d947b1b1c26da52d42ec66d66d00d95ae8d4c9b4f78
Sep 09 17:39:57.170207 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.170169    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq" event={"ID":"10e69cbb-2904-4ce0-a62b-ccac89a78922","Type":"ContainerStarted","Data":"064083543b9c49d09880ecc57d263125ca3c2b2b43e509b308993f77d6816bac"}
Sep 09 17:39:57.170659 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.170232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq" event={"ID":"10e69cbb-2904-4ce0-a62b-ccac89a78922","Type":"ContainerStarted","Data":"ec8fb963ee28ac114cfa4d947b1b1c26da52d42ec66d66d00d95ae8d4c9b4f78"}
Sep 09 17:39:57.170659 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.170258    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:39:57.171835 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.171804    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c" event={"ID":"4f6299fa-a3cb-49e4-84d2-448abadf0784","Type":"ContainerStarted","Data":"fbd727a0bab09ad757312cdfdcf2addc5c72e0f2cf9ed4f0e8fa42a3e3b9ad32"}
Sep 09 17:39:57.171943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.171860    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c" event={"ID":"4f6299fa-a3cb-49e4-84d2-448abadf0784","Type":"ContainerStarted","Data":"a2d1a8505685e3dbb2f4bf905fe0e9bfc90983be70ad103677c5c741e56bfc14"}
Sep 09 17:39:57.171943 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.171902    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:39:57.185558 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.185513    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq" podStartSLOduration=2.185499571 podStartE2EDuration="2.185499571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:57.184265207 +0000 UTC m=+1372.812417914" watchObservedRunningTime="2026-09-09 17:39:57.185499571 +0000 UTC m=+1372.813652357"
Sep 09 17:39:57.200920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:39:57.200868    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c" podStartSLOduration=2.200854078 podStartE2EDuration="2.200854078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:57.199488613 +0000 UTC m=+1372.827641316" watchObservedRunningTime="2026-09-09 17:39:57.200854078 +0000 UTC m=+1372.829006779"
Sep 09 17:40:04.933348 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:04.933320    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"]
Sep 09 17:40:04.938012 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:04.937993    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"
Sep 09 17:40:04.941049 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:04.940986    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:40:04.943130 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:04.943104    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"]
Sep 09 17:40:05.043664 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:05.043624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b3c10-aa67-4b93-aecf-7cc8574f4c39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk\" (UID: \"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"
Sep 09 17:40:05.144076 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:05.144037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b3c10-aa67-4b93-aecf-7cc8574f4c39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk\" (UID: \"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"
Sep 09 17:40:05.144684 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:05.144662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b3c10-aa67-4b93-aecf-7cc8574f4c39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk\" (UID: \"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"
Sep 09 17:40:05.252370 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:05.252296    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"
Sep 09 17:40:05.522532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:05.522503    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"]
Sep 09 17:40:05.524026 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:40:05.523992    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7b3c10_aa67_4b93_aecf_7cc8574f4c39.slice/crio-1d44c5c48818766efe3c7600f6698b40ce1cedce9efc55cfd6d1881659985b1b WatchSource:0}: Error finding container 1d44c5c48818766efe3c7600f6698b40ce1cedce9efc55cfd6d1881659985b1b: Status 404 returned error can't find the container with id 1d44c5c48818766efe3c7600f6698b40ce1cedce9efc55cfd6d1881659985b1b
Sep 09 17:40:06.224747 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:06.224712    2558 generic.go:356] "Generic (PLEG): container finished" podID="bc7b3c10-aa67-4b93-aecf-7cc8574f4c39" containerID="6fdb2cde7cd7aacb1b65c1b8b35cef4c2ce4dc050ffba75f00c0f98b2b9ff4df" exitCode=0
Sep 09 17:40:06.225154 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:06.224784    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk" event={"ID":"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39","Type":"ContainerDied","Data":"6fdb2cde7cd7aacb1b65c1b8b35cef4c2ce4dc050ffba75f00c0f98b2b9ff4df"}
Sep 09 17:40:06.225154 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:06.224832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk" event={"ID":"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39","Type":"ContainerStarted","Data":"1d44c5c48818766efe3c7600f6698b40ce1cedce9efc55cfd6d1881659985b1b"}
Sep 09 17:40:07.366627 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:07.366597    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"
Sep 09 17:40:07.465233 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:07.465187    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b3c10-aa67-4b93-aecf-7cc8574f4c39-maas-api-service-ca\") pod \"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39\" (UID: \"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39\") "
Sep 09 17:40:07.465568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:07.465543    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7b3c10-aa67-4b93-aecf-7cc8574f4c39-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc7b3c10-aa67-4b93-aecf-7cc8574f4c39" (UID: "bc7b3c10-aa67-4b93-aecf-7cc8574f4c39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:40:07.566092 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:07.566002    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b3c10-aa67-4b93-aecf-7cc8574f4c39-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:08.179547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:08.179520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:40:08.179716 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:08.179588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:40:08.241358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:08.241332    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"
Sep 09 17:40:08.241530 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:08.241370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk" event={"ID":"bc7b3c10-aa67-4b93-aecf-7cc8574f4c39","Type":"ContainerDied","Data":"1d44c5c48818766efe3c7600f6698b40ce1cedce9efc55cfd6d1881659985b1b"}
Sep 09 17:40:08.241530 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:08.241415    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d44c5c48818766efe3c7600f6698b40ce1cedce9efc55cfd6d1881659985b1b"
Sep 09 17:40:15.406862 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.406829    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"]
Sep 09 17:40:15.407346 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.407070    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67" podUID="bf12f5e8-08b6-4de5-9244-9c3b3677271b" containerName="payload-processing" containerID="cri-o://bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6" gracePeriod=30
Sep 09 17:40:15.410930 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.410893    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"]
Sep 09 17:40:15.411458 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.411413    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg" podUID="9e9016e1-79c5-48be-a043-4102fc23119c" containerName="payload-pre-processing" containerID="cri-o://f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe" gracePeriod=30
Sep 09 17:40:15.556487 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.556443    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"]
Sep 09 17:40:15.560310 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.560265    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-480517-pqtxk"]
Sep 09 17:40:15.684078 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.684048    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:40:15.686880 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.686861    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:40:15.737578 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.737539    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmhxj\" (UniqueName: \"kubernetes.io/projected/bf12f5e8-08b6-4de5-9244-9c3b3677271b-kube-api-access-bmhxj\") pod \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") "
Sep 09 17:40:15.737758 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.737598    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9016e1-79c5-48be-a043-4102fc23119c-plugins-config-volume\") pod \"9e9016e1-79c5-48be-a043-4102fc23119c\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") "
Sep 09 17:40:15.737758 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.737636    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xgft\" (UniqueName: \"kubernetes.io/projected/9e9016e1-79c5-48be-a043-4102fc23119c-kube-api-access-7xgft\") pod \"9e9016e1-79c5-48be-a043-4102fc23119c\" (UID: \"9e9016e1-79c5-48be-a043-4102fc23119c\") "
Sep 09 17:40:15.737758 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.737665    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf12f5e8-08b6-4de5-9244-9c3b3677271b-plugins-config-volume\") pod \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\" (UID: \"bf12f5e8-08b6-4de5-9244-9c3b3677271b\") "
Sep 09 17:40:15.738012 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.737981    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9016e1-79c5-48be-a043-4102fc23119c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e9016e1-79c5-48be-a043-4102fc23119c" (UID: "9e9016e1-79c5-48be-a043-4102fc23119c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:40:15.738124 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.738102    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf12f5e8-08b6-4de5-9244-9c3b3677271b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf12f5e8-08b6-4de5-9244-9c3b3677271b" (UID: "bf12f5e8-08b6-4de5-9244-9c3b3677271b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:40:15.739751 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.739725    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9016e1-79c5-48be-a043-4102fc23119c-kube-api-access-7xgft" (OuterVolumeSpecName: "kube-api-access-7xgft") pod "9e9016e1-79c5-48be-a043-4102fc23119c" (UID: "9e9016e1-79c5-48be-a043-4102fc23119c"). InnerVolumeSpecName "kube-api-access-7xgft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:40:15.740155 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.740135    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf12f5e8-08b6-4de5-9244-9c3b3677271b-kube-api-access-bmhxj" (OuterVolumeSpecName: "kube-api-access-bmhxj") pod "bf12f5e8-08b6-4de5-9244-9c3b3677271b" (UID: "bf12f5e8-08b6-4de5-9244-9c3b3677271b"). InnerVolumeSpecName "kube-api-access-bmhxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:40:15.839496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.839345    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmhxj\" (UniqueName: \"kubernetes.io/projected/bf12f5e8-08b6-4de5-9244-9c3b3677271b-kube-api-access-bmhxj\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:15.839496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.839383    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9016e1-79c5-48be-a043-4102fc23119c-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:15.839496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.839398    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xgft\" (UniqueName: \"kubernetes.io/projected/9e9016e1-79c5-48be-a043-4102fc23119c-kube-api-access-7xgft\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:15.839496 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:15.839414    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf12f5e8-08b6-4de5-9244-9c3b3677271b-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:16.286913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.286875    2558 generic.go:356] "Generic (PLEG): container finished" podID="bf12f5e8-08b6-4de5-9244-9c3b3677271b" containerID="bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6" exitCode=0
Sep 09 17:40:16.287081 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.286928    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"
Sep 09 17:40:16.287081 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.286976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67" event={"ID":"bf12f5e8-08b6-4de5-9244-9c3b3677271b","Type":"ContainerDied","Data":"bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6"}
Sep 09 17:40:16.287081 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.287028    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67" event={"ID":"bf12f5e8-08b6-4de5-9244-9c3b3677271b","Type":"ContainerDied","Data":"44f70d5c4bf8f9e97efaa3f8f088313ae8c5a3a289452513c0f589c51c3155e7"}
Sep 09 17:40:16.287081 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.287050    2558 scope.go:117] "RemoveContainer" containerID="bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6"
Sep 09 17:40:16.288533 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.288506    2558 generic.go:356] "Generic (PLEG): container finished" podID="9e9016e1-79c5-48be-a043-4102fc23119c" containerID="f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe" exitCode=0
Sep 09 17:40:16.288643 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.288564    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"
Sep 09 17:40:16.288643 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.288592    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg" event={"ID":"9e9016e1-79c5-48be-a043-4102fc23119c","Type":"ContainerDied","Data":"f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe"}
Sep 09 17:40:16.288723 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.288647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg" event={"ID":"9e9016e1-79c5-48be-a043-4102fc23119c","Type":"ContainerDied","Data":"4ca9860d3d3e6ecf2abbc6d269e8d1c38684f03ef636f1f745ff50ae7d752fc3"}
Sep 09 17:40:16.297081 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.297059    2558 scope.go:117] "RemoveContainer" containerID="bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6"
Sep 09 17:40:16.297479 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:40:16.297451    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6\": container with ID starting with bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6 not found: ID does not exist" containerID="bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6"
Sep 09 17:40:16.297591 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.297491    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6"} err="failed to get container status \"bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6\": rpc error: code = NotFound desc = could not find container \"bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6\": container with ID starting with bee40208c52dbc1a9b82f4266d8d014fd54786c6a5fa4ff4b66a890087a290f6 not found: ID does not exist"
Sep 09 17:40:16.297591 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.297517    2558 scope.go:117] "RemoveContainer" containerID="f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe"
Sep 09 17:40:16.306688 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.306669    2558 scope.go:117] "RemoveContainer" containerID="f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe"
Sep 09 17:40:16.307038 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:40:16.307011    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe\": container with ID starting with f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe not found: ID does not exist" containerID="f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe"
Sep 09 17:40:16.307100 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.307051    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe"} err="failed to get container status \"f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe\": rpc error: code = NotFound desc = could not find container \"f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe\": container with ID starting with f8a214758abdf9d3b68cd9a59e3cb77ec8f44861b2a5f982bd734749b936fcfe not found: ID does not exist"
Sep 09 17:40:16.312684 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.312655    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"]
Sep 09 17:40:16.316354 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.316329    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-480517-69f55b5fcc-f7b67"]
Sep 09 17:40:16.325995 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.325945    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"]
Sep 09 17:40:16.330361 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.330331    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-480517-57bc8848b-d85pg"]
Sep 09 17:40:16.816372 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.816339    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e9016e1-79c5-48be-a043-4102fc23119c" path="/var/lib/kubelet/pods/9e9016e1-79c5-48be-a043-4102fc23119c/volumes"
Sep 09 17:40:16.816780 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.816758    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7b3c10-aa67-4b93-aecf-7cc8574f4c39" path="/var/lib/kubelet/pods/bc7b3c10-aa67-4b93-aecf-7cc8574f4c39/volumes"
Sep 09 17:40:16.817128 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:16.817114    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf12f5e8-08b6-4de5-9244-9c3b3677271b" path="/var/lib/kubelet/pods/bf12f5e8-08b6-4de5-9244-9c3b3677271b/volumes"
Sep 09 17:40:22.178653 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.178615    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"]
Sep 09 17:40:22.179054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.178948    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7b3c10-aa67-4b93-aecf-7cc8574f4c39" containerName="revoke-keys"
Sep 09 17:40:22.179054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.178983    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7b3c10-aa67-4b93-aecf-7cc8574f4c39" containerName="revoke-keys"
Sep 09 17:40:22.179054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.178992    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9016e1-79c5-48be-a043-4102fc23119c" containerName="payload-pre-processing"
Sep 09 17:40:22.179054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.178997    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9016e1-79c5-48be-a043-4102fc23119c" containerName="payload-pre-processing"
Sep 09 17:40:22.179054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.179014    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf12f5e8-08b6-4de5-9244-9c3b3677271b" containerName="payload-processing"
Sep 09 17:40:22.179054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.179020    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf12f5e8-08b6-4de5-9244-9c3b3677271b" containerName="payload-processing"
Sep 09 17:40:22.179276 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.179083    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9016e1-79c5-48be-a043-4102fc23119c" containerName="payload-pre-processing"
Sep 09 17:40:22.179276 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.179111    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7b3c10-aa67-4b93-aecf-7cc8574f4c39" containerName="revoke-keys"
Sep 09 17:40:22.179276 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.179120    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf12f5e8-08b6-4de5-9244-9c3b3677271b" containerName="payload-processing"
Sep 09 17:40:22.181883 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.181866    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"
Sep 09 17:40:22.184381 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.184360    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:40:22.185921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.185890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a69a7d-4853-470a-857b-f3b2c90159e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk\" (UID: \"d3a69a7d-4853-470a-857b-f3b2c90159e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"
Sep 09 17:40:22.188737 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.188712    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"]
Sep 09 17:40:22.287047 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.287007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a69a7d-4853-470a-857b-f3b2c90159e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk\" (UID: \"d3a69a7d-4853-470a-857b-f3b2c90159e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"
Sep 09 17:40:22.287635 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.287613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a69a7d-4853-470a-857b-f3b2c90159e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk\" (UID: \"d3a69a7d-4853-470a-857b-f3b2c90159e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"
Sep 09 17:40:22.493939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.493849    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"
Sep 09 17:40:22.766391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:22.766311    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"]
Sep 09 17:40:22.769817 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:40:22.769781    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a69a7d_4853_470a_857b_f3b2c90159e0.slice/crio-375acb26373c6ac8e8e8797b4f58d5242532a587160124a5e670d6a57e07f15d WatchSource:0}: Error finding container 375acb26373c6ac8e8e8797b4f58d5242532a587160124a5e670d6a57e07f15d: Status 404 returned error can't find the container with id 375acb26373c6ac8e8e8797b4f58d5242532a587160124a5e670d6a57e07f15d
Sep 09 17:40:23.334941 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:23.334908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk" event={"ID":"d3a69a7d-4853-470a-857b-f3b2c90159e0","Type":"ContainerStarted","Data":"9ababa3f6db2c26dab6c1e00376caf9d207f12d4a9807823cabcc9b98b961885"}
Sep 09 17:40:23.335357 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:23.334980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk" event={"ID":"d3a69a7d-4853-470a-857b-f3b2c90159e0","Type":"ContainerStarted","Data":"375acb26373c6ac8e8e8797b4f58d5242532a587160124a5e670d6a57e07f15d"}
Sep 09 17:40:23.350920 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:23.350794    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk" podStartSLOduration=1.350779236 podStartE2EDuration="1.350779236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:40:23.349819412 +0000 UTC m=+1398.977972116" watchObservedRunningTime="2026-09-09 17:40:23.350779236 +0000 UTC m=+1398.978931978"
Sep 09 17:40:24.341901 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:24.341865    2558 generic.go:356] "Generic (PLEG): container finished" podID="d3a69a7d-4853-470a-857b-f3b2c90159e0" containerID="9ababa3f6db2c26dab6c1e00376caf9d207f12d4a9807823cabcc9b98b961885" exitCode=0
Sep 09 17:40:24.342344 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:24.341963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk" event={"ID":"d3a69a7d-4853-470a-857b-f3b2c90159e0","Type":"ContainerDied","Data":"9ababa3f6db2c26dab6c1e00376caf9d207f12d4a9807823cabcc9b98b961885"}
Sep 09 17:40:25.476458 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:25.476433    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"
Sep 09 17:40:25.513770 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:25.513737    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a69a7d-4853-470a-857b-f3b2c90159e0-maas-api-service-ca\") pod \"d3a69a7d-4853-470a-857b-f3b2c90159e0\" (UID: \"d3a69a7d-4853-470a-857b-f3b2c90159e0\") "
Sep 09 17:40:25.514154 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:25.514128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a69a7d-4853-470a-857b-f3b2c90159e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3a69a7d-4853-470a-857b-f3b2c90159e0" (UID: "d3a69a7d-4853-470a-857b-f3b2c90159e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:40:25.615440 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:25.615353    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a69a7d-4853-470a-857b-f3b2c90159e0-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:26.354878 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:26.354848    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"
Sep 09 17:40:26.355089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:26.354892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk" event={"ID":"d3a69a7d-4853-470a-857b-f3b2c90159e0","Type":"ContainerDied","Data":"375acb26373c6ac8e8e8797b4f58d5242532a587160124a5e670d6a57e07f15d"}
Sep 09 17:40:26.355089 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:26.354942    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="375acb26373c6ac8e8e8797b4f58d5242532a587160124a5e670d6a57e07f15d"
Sep 09 17:40:32.711529 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:32.711462    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"]
Sep 09 17:40:32.712146 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:32.711763    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8" podUID="07e33598-3b60-42b9-bfc2-c282300a8d7d" containerName="payload-pre-processing" containerID="cri-o://aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856" gracePeriod=30
Sep 09 17:40:32.866484 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:32.866447    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"]
Sep 09 17:40:32.871217 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:32.871187    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17e67b-74pwk"]
Sep 09 17:40:32.963997 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:32.963919    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:40:33.080642 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.080601    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48pqv\" (UniqueName: \"kubernetes.io/projected/07e33598-3b60-42b9-bfc2-c282300a8d7d-kube-api-access-48pqv\") pod \"07e33598-3b60-42b9-bfc2-c282300a8d7d\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") "
Sep 09 17:40:33.080847 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.080712    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e33598-3b60-42b9-bfc2-c282300a8d7d-plugins-config-volume\") pod \"07e33598-3b60-42b9-bfc2-c282300a8d7d\" (UID: \"07e33598-3b60-42b9-bfc2-c282300a8d7d\") "
Sep 09 17:40:33.081104 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.081079    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e33598-3b60-42b9-bfc2-c282300a8d7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07e33598-3b60-42b9-bfc2-c282300a8d7d" (UID: "07e33598-3b60-42b9-bfc2-c282300a8d7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:40:33.082757 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.082736    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e33598-3b60-42b9-bfc2-c282300a8d7d-kube-api-access-48pqv" (OuterVolumeSpecName: "kube-api-access-48pqv") pod "07e33598-3b60-42b9-bfc2-c282300a8d7d" (UID: "07e33598-3b60-42b9-bfc2-c282300a8d7d"). InnerVolumeSpecName "kube-api-access-48pqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:40:33.182156 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.182112    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e33598-3b60-42b9-bfc2-c282300a8d7d-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:33.182156 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.182150    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48pqv\" (UniqueName: \"kubernetes.io/projected/07e33598-3b60-42b9-bfc2-c282300a8d7d-kube-api-access-48pqv\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:40:33.400978 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.400872    2558 generic.go:356] "Generic (PLEG): container finished" podID="07e33598-3b60-42b9-bfc2-c282300a8d7d" containerID="aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856" exitCode=0
Sep 09 17:40:33.400978 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.400914    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"
Sep 09 17:40:33.401154 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.400979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8" event={"ID":"07e33598-3b60-42b9-bfc2-c282300a8d7d","Type":"ContainerDied","Data":"aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856"}
Sep 09 17:40:33.401154 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.401037    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8" event={"ID":"07e33598-3b60-42b9-bfc2-c282300a8d7d","Type":"ContainerDied","Data":"2c2ed4cb26ac31a8510ca523b66ff0fd035df0ff131f91397ce50c77faf44a34"}
Sep 09 17:40:33.401154 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.401059    2558 scope.go:117] "RemoveContainer" containerID="aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856"
Sep 09 17:40:33.411270 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.411242    2558 scope.go:117] "RemoveContainer" containerID="aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856"
Sep 09 17:40:33.411604 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:40:33.411580    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856\": container with ID starting with aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856 not found: ID does not exist" containerID="aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856"
Sep 09 17:40:33.411665 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.411613    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856"} err="failed to get container status \"aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856\": rpc error: code = NotFound desc = could not find container \"aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856\": container with ID starting with aab39f4c56e6b53577d276812c46c89ca2defe7b51343e5d072e360d78e53856 not found: ID does not exist"
Sep 09 17:40:33.422717 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.422687    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"]
Sep 09 17:40:33.431650 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:33.431620    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-17e67b-7799cff4cd-j6vl8"]
Sep 09 17:40:34.817724 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:34.817688    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e33598-3b60-42b9-bfc2-c282300a8d7d" path="/var/lib/kubelet/pods/07e33598-3b60-42b9-bfc2-c282300a8d7d/volumes"
Sep 09 17:40:34.818213 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:34.818182    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a69a7d-4853-470a-857b-f3b2c90159e0" path="/var/lib/kubelet/pods/d3a69a7d-4853-470a-857b-f3b2c90159e0/volumes"
Sep 09 17:40:36.842933 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.842899    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"]
Sep 09 17:40:36.843358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.843267    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e33598-3b60-42b9-bfc2-c282300a8d7d" containerName="payload-pre-processing"
Sep 09 17:40:36.843358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.843279    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e33598-3b60-42b9-bfc2-c282300a8d7d" containerName="payload-pre-processing"
Sep 09 17:40:36.843358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.843294    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a69a7d-4853-470a-857b-f3b2c90159e0" containerName="revoke-keys"
Sep 09 17:40:36.843358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.843299    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a69a7d-4853-470a-857b-f3b2c90159e0" containerName="revoke-keys"
Sep 09 17:40:36.843358 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.843358    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a69a7d-4853-470a-857b-f3b2c90159e0" containerName="revoke-keys"
Sep 09 17:40:36.843526 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.843368    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e33598-3b60-42b9-bfc2-c282300a8d7d" containerName="payload-pre-processing"
Sep 09 17:40:36.847314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.847283    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:36.850662 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.850433    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f636a1c6-dockercfg-lb8p2\""
Sep 09 17:40:36.850662 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.850608    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f636a1c6\""
Sep 09 17:40:36.856433 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.856052    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"]
Sep 09 17:40:36.915603 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.915561    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f9ba92-cf7c-4b02-b488-df0a088c4eee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:36.915811 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:36.915646    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5fl\" (UniqueName: \"kubernetes.io/projected/70f9ba92-cf7c-4b02-b488-df0a088c4eee-kube-api-access-qz5fl\") pod \"payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:37.016809 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:37.016767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f9ba92-cf7c-4b02-b488-df0a088c4eee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:37.016809 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:37.016814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5fl\" (UniqueName: \"kubernetes.io/projected/70f9ba92-cf7c-4b02-b488-df0a088c4eee-kube-api-access-qz5fl\") pod \"payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:37.017526 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:37.017504    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f9ba92-cf7c-4b02-b488-df0a088c4eee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:37.024733 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:37.024703    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5fl\" (UniqueName: \"kubernetes.io/projected/70f9ba92-cf7c-4b02-b488-df0a088c4eee-kube-api-access-qz5fl\") pod \"payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:37.160407 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:37.160311    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:37.436720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:37.436682    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"]
Sep 09 17:40:37.437861 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:40:37.437831    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f9ba92_cf7c_4b02_b488_df0a088c4eee.slice/crio-667d238f185722a6ab8e2278202232fe5972af1d1408b9d912faa271206387ce WatchSource:0}: Error finding container 667d238f185722a6ab8e2278202232fe5972af1d1408b9d912faa271206387ce: Status 404 returned error can't find the container with id 667d238f185722a6ab8e2278202232fe5972af1d1408b9d912faa271206387ce
Sep 09 17:40:38.432370 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:38.432334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb" event={"ID":"70f9ba92-cf7c-4b02-b488-df0a088c4eee","Type":"ContainerStarted","Data":"55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623"}
Sep 09 17:40:38.432764 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:38.432393    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb" event={"ID":"70f9ba92-cf7c-4b02-b488-df0a088c4eee","Type":"ContainerStarted","Data":"667d238f185722a6ab8e2278202232fe5972af1d1408b9d912faa271206387ce"}
Sep 09 17:40:38.432764 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:38.432421    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:40:38.461225 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:38.461167    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb" podStartSLOduration=2.461149969 podStartE2EDuration="2.461149969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:40:38.45832315 +0000 UTC m=+1414.086475854" watchObservedRunningTime="2026-09-09 17:40:38.461149969 +0000 UTC m=+1414.089302671"
Sep 09 17:40:49.440344 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:40:49.440310    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:41:04.296670 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.296586    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"]
Sep 09 17:41:04.300352 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.300309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.303038 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.303014    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f636a1c6\"/\"default-dockercfg-l8qn2\""
Sep 09 17:41:04.303205 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.303018    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f636a1c6\"/\"kube-root-ca.crt\""
Sep 09 17:41:04.303205 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.303019    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f636a1c6\"/\"openshift-service-ca.crt\""
Sep 09 17:41:04.304351 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.304327    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f636a1c6\"/\"e2e-lifecycle-model-f636a1c6-kserve-self-signed-certs\""
Sep 09 17:41:04.307521 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.307220    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"]
Sep 09 17:41:04.460182 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.460141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-dshm\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.460375 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.460265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tmp-dir\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.460375 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.460298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tls-certs\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.460375 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.460352    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwm2h\" (UniqueName: \"kubernetes.io/projected/f2e005fb-8f82-4575-b592-f3f0d92a98d6-kube-api-access-bwm2h\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.460480 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.460380    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-home\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.460480 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.460397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-model-cache\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.560987 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.560882    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-home\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.560987 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.560917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-model-cache\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.560987 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.560941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-dshm\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.561265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.561017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tmp-dir\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.561265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.561033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tls-certs\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.561265 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.561064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm2h\" (UniqueName: \"kubernetes.io/projected/f2e005fb-8f82-4575-b592-f3f0d92a98d6-kube-api-access-bwm2h\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.561428 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.561357    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-home\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.561428 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.561408    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-model-cache\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.561535 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.561443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tmp-dir\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.563294 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.563269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-dshm\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.563527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.563511    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tls-certs\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.568766 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.568740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwm2h\" (UniqueName: \"kubernetes.io/projected/f2e005fb-8f82-4575-b592-f3f0d92a98d6-kube-api-access-bwm2h\") pod \"e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") " pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.610901 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.610870    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:04.881323 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:04.881292    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"]
Sep 09 17:41:04.882527 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:41:04.882502    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2e005fb_8f82_4575_b592_f3f0d92a98d6.slice/crio-524304354ceda509a15d7c46f46ee75ffe1143a645b4b9aeedce12487bd75bbc WatchSource:0}: Error finding container 524304354ceda509a15d7c46f46ee75ffe1143a645b4b9aeedce12487bd75bbc: Status 404 returned error can't find the container with id 524304354ceda509a15d7c46f46ee75ffe1143a645b4b9aeedce12487bd75bbc
Sep 09 17:41:05.583024 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:05.582986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz" event={"ID":"f2e005fb-8f82-4575-b592-f3f0d92a98d6","Type":"ContainerStarted","Data":"b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f"}
Sep 09 17:41:05.583484 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:05.583034    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz" event={"ID":"f2e005fb-8f82-4575-b592-f3f0d92a98d6","Type":"ContainerStarted","Data":"524304354ceda509a15d7c46f46ee75ffe1143a645b4b9aeedce12487bd75bbc"}
Sep 09 17:41:05.599024 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:05.598974    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz" podStartSLOduration=1.598943857 podStartE2EDuration="1.598943857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:41:05.597739598 +0000 UTC m=+1441.225892299" watchObservedRunningTime="2026-09-09 17:41:05.598943857 +0000 UTC m=+1441.227096559"
Sep 09 17:41:14.611944 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:14.611909    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:14.612362 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:14.611973    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:14.624446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:14.624414    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:14.645204 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:14.645176    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:41:24.037582 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.037542    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"]
Sep 09 17:41:24.041127 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.041109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"
Sep 09 17:41:24.043506 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.043483    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:41:24.048087 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.048060    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"]
Sep 09 17:41:24.121828 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.121789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae031432-58b1-471f-aec5-e5f6a4b12a85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg\" (UID: \"ae031432-58b1-471f-aec5-e5f6a4b12a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"
Sep 09 17:41:24.223459 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.223416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae031432-58b1-471f-aec5-e5f6a4b12a85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg\" (UID: \"ae031432-58b1-471f-aec5-e5f6a4b12a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"
Sep 09 17:41:24.224136 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.224118    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae031432-58b1-471f-aec5-e5f6a4b12a85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg\" (UID: \"ae031432-58b1-471f-aec5-e5f6a4b12a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"
Sep 09 17:41:24.353193 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.353110    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"
Sep 09 17:41:24.625469 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.625446    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"]
Sep 09 17:41:24.627233 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:41:24.627206    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae031432_58b1_471f_aec5_e5f6a4b12a85.slice/crio-17b6d8546cb8761ce1a6c70b3bd5d78039934bb9e191a49f6dd072a81060c72d WatchSource:0}: Error finding container 17b6d8546cb8761ce1a6c70b3bd5d78039934bb9e191a49f6dd072a81060c72d: Status 404 returned error can't find the container with id 17b6d8546cb8761ce1a6c70b3bd5d78039934bb9e191a49f6dd072a81060c72d
Sep 09 17:41:24.688723 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:24.688687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg" event={"ID":"ae031432-58b1-471f-aec5-e5f6a4b12a85","Type":"ContainerStarted","Data":"17b6d8546cb8761ce1a6c70b3bd5d78039934bb9e191a49f6dd072a81060c72d"}
Sep 09 17:41:25.695985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:25.695931    2558 generic.go:356] "Generic (PLEG): container finished" podID="ae031432-58b1-471f-aec5-e5f6a4b12a85" containerID="e5363a5f6006dcf146ba99b446a5baf0ba51c154c5f9d30ed26fd191a1892e5f" exitCode=0
Sep 09 17:41:25.696385 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:25.696024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg" event={"ID":"ae031432-58b1-471f-aec5-e5f6a4b12a85","Type":"ContainerDied","Data":"e5363a5f6006dcf146ba99b446a5baf0ba51c154c5f9d30ed26fd191a1892e5f"}
Sep 09 17:41:26.832232 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:26.832206    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"
Sep 09 17:41:26.948341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:26.948305    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae031432-58b1-471f-aec5-e5f6a4b12a85-maas-api-service-ca\") pod \"ae031432-58b1-471f-aec5-e5f6a4b12a85\" (UID: \"ae031432-58b1-471f-aec5-e5f6a4b12a85\") "
Sep 09 17:41:26.948708 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:26.948679    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae031432-58b1-471f-aec5-e5f6a4b12a85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae031432-58b1-471f-aec5-e5f6a4b12a85" (UID: "ae031432-58b1-471f-aec5-e5f6a4b12a85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:41:27.049916 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:27.049827    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae031432-58b1-471f-aec5-e5f6a4b12a85-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:41:27.708470 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:27.708430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg" event={"ID":"ae031432-58b1-471f-aec5-e5f6a4b12a85","Type":"ContainerDied","Data":"17b6d8546cb8761ce1a6c70b3bd5d78039934bb9e191a49f6dd072a81060c72d"}
Sep 09 17:41:27.708470 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:27.708453    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"
Sep 09 17:41:27.708683 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:27.708488    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b6d8546cb8761ce1a6c70b3bd5d78039934bb9e191a49f6dd072a81060c72d"
Sep 09 17:41:39.868564 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:39.868507    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"]
Sep 09 17:41:39.869348 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:39.868857    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb" podUID="70f9ba92-cf7c-4b02-b488-df0a088c4eee" containerName="payload-pre-processing" containerID="cri-o://55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623" gracePeriod=30
Sep 09 17:41:40.020539 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.020503    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"]
Sep 09 17:41:40.022646 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.022615    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f636a1c6-mdddg"]
Sep 09 17:41:40.120745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.120686    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:41:40.171658 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.171626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f9ba92-cf7c-4b02-b488-df0a088c4eee-plugins-config-volume\") pod \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") "
Sep 09 17:41:40.171819 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.171685    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz5fl\" (UniqueName: \"kubernetes.io/projected/70f9ba92-cf7c-4b02-b488-df0a088c4eee-kube-api-access-qz5fl\") pod \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\" (UID: \"70f9ba92-cf7c-4b02-b488-df0a088c4eee\") "
Sep 09 17:41:40.172038 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.172014    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f9ba92-cf7c-4b02-b488-df0a088c4eee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70f9ba92-cf7c-4b02-b488-df0a088c4eee" (UID: "70f9ba92-cf7c-4b02-b488-df0a088c4eee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:41:40.173693 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.173671    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f9ba92-cf7c-4b02-b488-df0a088c4eee-kube-api-access-qz5fl" (OuterVolumeSpecName: "kube-api-access-qz5fl") pod "70f9ba92-cf7c-4b02-b488-df0a088c4eee" (UID: "70f9ba92-cf7c-4b02-b488-df0a088c4eee"). InnerVolumeSpecName "kube-api-access-qz5fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:41:40.272450 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.272401    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz5fl\" (UniqueName: \"kubernetes.io/projected/70f9ba92-cf7c-4b02-b488-df0a088c4eee-kube-api-access-qz5fl\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:41:40.272450 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.272441    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f9ba92-cf7c-4b02-b488-df0a088c4eee-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:41:40.783341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.783303    2558 generic.go:356] "Generic (PLEG): container finished" podID="70f9ba92-cf7c-4b02-b488-df0a088c4eee" containerID="55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623" exitCode=0
Sep 09 17:41:40.783525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.783350    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"
Sep 09 17:41:40.783525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.783359    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb" event={"ID":"70f9ba92-cf7c-4b02-b488-df0a088c4eee","Type":"ContainerDied","Data":"55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623"}
Sep 09 17:41:40.783525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.783414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb" event={"ID":"70f9ba92-cf7c-4b02-b488-df0a088c4eee","Type":"ContainerDied","Data":"667d238f185722a6ab8e2278202232fe5972af1d1408b9d912faa271206387ce"}
Sep 09 17:41:40.783525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.783445    2558 scope.go:117] "RemoveContainer" containerID="55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623"
Sep 09 17:41:40.792812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.792786    2558 scope.go:117] "RemoveContainer" containerID="55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623"
Sep 09 17:41:40.793125 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:41:40.793104    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623\": container with ID starting with 55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623 not found: ID does not exist" containerID="55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623"
Sep 09 17:41:40.793200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.793135    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623"} err="failed to get container status \"55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623\": rpc error: code = NotFound desc = could not find container \"55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623\": container with ID starting with 55038e1692b500b26ea40df4b86b2c01bd397c460e4425a80425b3db02433623 not found: ID does not exist"
Sep 09 17:41:40.807431 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.807401    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"]
Sep 09 17:41:40.816733 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.816699    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae031432-58b1-471f-aec5-e5f6a4b12a85" path="/var/lib/kubelet/pods/ae031432-58b1-471f-aec5-e5f6a4b12a85/volumes"
Sep 09 17:41:40.817016 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:40.817001    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f636a1c6-7b444cd9fd-m8tmb"]
Sep 09 17:41:42.816855 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:42.816823    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f9ba92-cf7c-4b02-b488-df0a088c4eee" path="/var/lib/kubelet/pods/70f9ba92-cf7c-4b02-b488-df0a088c4eee/volumes"
Sep 09 17:41:48.418718 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:41:48.418676    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"]
Sep 09 17:42:03.439294 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.439227    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz" podUID="f2e005fb-8f82-4575-b592-f3f0d92a98d6" containerName="main" containerID="cri-o://b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f" gracePeriod=45
Sep 09 17:42:03.700095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.700021    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:42:03.874304 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-home\") pod \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") "
Sep 09 17:42:03.874473 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874324    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-dshm\") pod \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") "
Sep 09 17:42:03.874473 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tmp-dir\") pod \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") "
Sep 09 17:42:03.874473 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874424    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tls-certs\") pod \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") "
Sep 09 17:42:03.874627 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874475    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwm2h\" (UniqueName: \"kubernetes.io/projected/f2e005fb-8f82-4575-b592-f3f0d92a98d6-kube-api-access-bwm2h\") pod \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") "
Sep 09 17:42:03.874627 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874534    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-model-cache\") pod \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\" (UID: \"f2e005fb-8f82-4575-b592-f3f0d92a98d6\") "
Sep 09 17:42:03.874627 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874610    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-home" (OuterVolumeSpecName: "home") pod "f2e005fb-8f82-4575-b592-f3f0d92a98d6" (UID: "f2e005fb-8f82-4575-b592-f3f0d92a98d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:42:03.874778 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874713    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2e005fb-8f82-4575-b592-f3f0d92a98d6" (UID: "f2e005fb-8f82-4575-b592-f3f0d92a98d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:42:03.874854 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874834    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2e005fb-8f82-4575-b592-f3f0d92a98d6" (UID: "f2e005fb-8f82-4575-b592-f3f0d92a98d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:42:03.874913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874850    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:42:03.874913 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.874870    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:42:03.876670 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.876646    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-dshm" (OuterVolumeSpecName: "dshm") pod "f2e005fb-8f82-4575-b592-f3f0d92a98d6" (UID: "f2e005fb-8f82-4575-b592-f3f0d92a98d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:42:03.876670 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.876659    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e005fb-8f82-4575-b592-f3f0d92a98d6-kube-api-access-bwm2h" (OuterVolumeSpecName: "kube-api-access-bwm2h") pod "f2e005fb-8f82-4575-b592-f3f0d92a98d6" (UID: "f2e005fb-8f82-4575-b592-f3f0d92a98d6"). InnerVolumeSpecName "kube-api-access-bwm2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:42:03.876892 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.876683    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2e005fb-8f82-4575-b592-f3f0d92a98d6" (UID: "f2e005fb-8f82-4575-b592-f3f0d92a98d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:42:03.919043 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.919009    2558 generic.go:356] "Generic (PLEG): container finished" podID="f2e005fb-8f82-4575-b592-f3f0d92a98d6" containerID="b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f" exitCode=0
Sep 09 17:42:03.919225 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.919058    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"
Sep 09 17:42:03.919225 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.919089    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz" event={"ID":"f2e005fb-8f82-4575-b592-f3f0d92a98d6","Type":"ContainerDied","Data":"b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f"}
Sep 09 17:42:03.919225 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.919128    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz" event={"ID":"f2e005fb-8f82-4575-b592-f3f0d92a98d6","Type":"ContainerDied","Data":"524304354ceda509a15d7c46f46ee75ffe1143a645b4b9aeedce12487bd75bbc"}
Sep 09 17:42:03.919225 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.919145    2558 scope.go:117] "RemoveContainer" containerID="b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f"
Sep 09 17:42:03.928354 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.928339    2558 scope.go:117] "RemoveContainer" containerID="b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f"
Sep 09 17:42:03.928616 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:42:03.928599    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f\": container with ID starting with b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f not found: ID does not exist" containerID="b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f"
Sep 09 17:42:03.928656 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.928624    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f"} err="failed to get container status \"b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f\": rpc error: code = NotFound desc = could not find container \"b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f\": container with ID starting with b49f78db14df39b2d5f98252fe1abcf6b9bd9536b9f42e12c44511163e66551f not found: ID does not exist"
Sep 09 17:42:03.942391 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.942361    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"]
Sep 09 17:42:03.948527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.948501    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f636a1c6/e2e-lifecycle-model-f636a1c6-kserve-85ffdb4b55-8t8rz"]
Sep 09 17:42:03.975778 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.975690    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:42:03.975778 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.975726    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e005fb-8f82-4575-b592-f3f0d92a98d6-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:42:03.975778 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.975743    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwm2h\" (UniqueName: \"kubernetes.io/projected/f2e005fb-8f82-4575-b592-f3f0d92a98d6-kube-api-access-bwm2h\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:42:03.975778 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:03.975757    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e005fb-8f82-4575-b592-f3f0d92a98d6-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:42:04.818214 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:04.818176    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e005fb-8f82-4575-b592-f3f0d92a98d6" path="/var/lib/kubelet/pods/f2e005fb-8f82-4575-b592-f3f0d92a98d6/volumes"
Sep 09 17:42:04.836438 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:04.836408    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:42:04.838907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:04.838886    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:42:04.960648 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:04.960617    2558 scope.go:117] "RemoveContainer" containerID="b7bc7ab238d3a15c492699bb46dd6a157983f330c3db97977a75d880654e3eb2"
Sep 09 17:42:04.968803 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:42:04.968782    2558 scope.go:117] "RemoveContainer" containerID="ce0e76d4779ca7f926503aa8368e2c1b871ae1ae61ebbec0351dabffbc006ef4"
Sep 09 17:43:04.999553 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:04.999521    2558 scope.go:117] "RemoveContainer" containerID="f2b20b0570a2f78840bc2f31444761487bf4870148ff714b3edacb40bda3fa28"
Sep 09 17:43:05.010200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:05.010183    2558 scope.go:117] "RemoveContainer" containerID="5c3f41409b654494c5e1c85c47fc8db445d8acfb916b6d749ddb13698d8ce4b8"
Sep 09 17:43:35.709004 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.708963    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"]
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709295    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e005fb-8f82-4575-b592-f3f0d92a98d6" containerName="main"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709307    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e005fb-8f82-4575-b592-f3f0d92a98d6" containerName="main"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709315    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae031432-58b1-471f-aec5-e5f6a4b12a85" containerName="revoke-keys"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709321    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae031432-58b1-471f-aec5-e5f6a4b12a85" containerName="revoke-keys"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709330    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f9ba92-cf7c-4b02-b488-df0a088c4eee" containerName="payload-pre-processing"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709336    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f9ba92-cf7c-4b02-b488-df0a088c4eee" containerName="payload-pre-processing"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709389    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e005fb-8f82-4575-b592-f3f0d92a98d6" containerName="main"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709397    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f9ba92-cf7c-4b02-b488-df0a088c4eee" containerName="payload-pre-processing"
Sep 09 17:43:35.709426 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.709403    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae031432-58b1-471f-aec5-e5f6a4b12a85" containerName="revoke-keys"
Sep 09 17:43:35.712177 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.712159    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:35.714753 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.714724    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cf91eb-dockercfg-rhkt7\""
Sep 09 17:43:35.714900 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.714828    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cf91eb\""
Sep 09 17:43:35.720696 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.720669    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"]
Sep 09 17:43:35.880991 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.880943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9212d022-7d83-4d21-be11-4b43e0d0aaf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:35.880991 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.880998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9f9\" (UniqueName: \"kubernetes.io/projected/9212d022-7d83-4d21-be11-4b43e0d0aaf9-kube-api-access-dx9f9\") pod \"payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:35.982517 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.982429    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9212d022-7d83-4d21-be11-4b43e0d0aaf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:35.982517 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.982472    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9f9\" (UniqueName: \"kubernetes.io/projected/9212d022-7d83-4d21-be11-4b43e0d0aaf9-kube-api-access-dx9f9\") pod \"payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:35.983297 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.983270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9212d022-7d83-4d21-be11-4b43e0d0aaf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:35.990334 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:35.990306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9f9\" (UniqueName: \"kubernetes.io/projected/9212d022-7d83-4d21-be11-4b43e0d0aaf9-kube-api-access-dx9f9\") pod \"payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:36.023530 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:36.023492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:36.297239 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:36.297208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"]
Sep 09 17:43:36.299076 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:43:36.299050    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9212d022_7d83_4d21_be11_4b43e0d0aaf9.slice/crio-dc4faaf2f630cd9c5665d4a98e5de8f45a8536d449bf45bea91dc06f96690c4e WatchSource:0}: Error finding container dc4faaf2f630cd9c5665d4a98e5de8f45a8536d449bf45bea91dc06f96690c4e: Status 404 returned error can't find the container with id dc4faaf2f630cd9c5665d4a98e5de8f45a8536d449bf45bea91dc06f96690c4e
Sep 09 17:43:36.415125 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:36.415085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g" event={"ID":"9212d022-7d83-4d21-be11-4b43e0d0aaf9","Type":"ContainerStarted","Data":"7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c"}
Sep 09 17:43:36.415306 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:36.415151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g" event={"ID":"9212d022-7d83-4d21-be11-4b43e0d0aaf9","Type":"ContainerStarted","Data":"dc4faaf2f630cd9c5665d4a98e5de8f45a8536d449bf45bea91dc06f96690c4e"}
Sep 09 17:43:36.415306 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:36.415278    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:43:36.430643 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:36.430587    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g" podStartSLOduration=1.430566995 podStartE2EDuration="1.430566995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:43:36.429140413 +0000 UTC m=+1592.057293119" watchObservedRunningTime="2026-09-09 17:43:36.430566995 +0000 UTC m=+1592.058719697"
Sep 09 17:43:47.422828 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:43:47.422797    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:44:04.881246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.881172    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"]
Sep 09 17:44:04.884715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.884699    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:04.887305 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.887285    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ecf58e\""
Sep 09 17:44:04.893333 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.893309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"]
Sep 09 17:44:04.926933 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.926904    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"]
Sep 09 17:44:04.928976 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.928944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172810f8-62cb-4db5-842b-b56547ec67c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:04.929056 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.929038    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcsh\" (UniqueName: \"kubernetes.io/projected/172810f8-62cb-4db5-842b-b56547ec67c7-kube-api-access-6hcsh\") pod \"payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:04.930219 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.930204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:04.932811 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.932797    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ecf58e-dockercfg-hz85x\""
Sep 09 17:44:04.940005 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:04.939943    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"]
Sep 09 17:44:05.030134 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.030101    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtc8k\" (UniqueName: \"kubernetes.io/projected/8af953a9-987d-4f15-b265-0c66b6fc5085-kube-api-access-qtc8k\") pod \"payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:05.030338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.030157    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172810f8-62cb-4db5-842b-b56547ec67c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:05.030338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.030242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcsh\" (UniqueName: \"kubernetes.io/projected/172810f8-62cb-4db5-842b-b56547ec67c7-kube-api-access-6hcsh\") pod \"payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:05.030338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.030271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af953a9-987d-4f15-b265-0c66b6fc5085-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:05.030886 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.030864    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172810f8-62cb-4db5-842b-b56547ec67c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:05.039126 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.039096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcsh\" (UniqueName: \"kubernetes.io/projected/172810f8-62cb-4db5-842b-b56547ec67c7-kube-api-access-6hcsh\") pod \"payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:05.131016 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.130974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtc8k\" (UniqueName: \"kubernetes.io/projected/8af953a9-987d-4f15-b265-0c66b6fc5085-kube-api-access-qtc8k\") pod \"payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:05.131190 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.131075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af953a9-987d-4f15-b265-0c66b6fc5085-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:05.131720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.131676    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af953a9-987d-4f15-b265-0c66b6fc5085-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:05.138470 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.138448    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtc8k\" (UniqueName: \"kubernetes.io/projected/8af953a9-987d-4f15-b265-0c66b6fc5085-kube-api-access-qtc8k\") pod \"payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:05.195821 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.195784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:05.241040 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.241005    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:05.470665 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.470632    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"]
Sep 09 17:44:05.472633 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:44:05.472606    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172810f8_62cb_4db5_842b_b56547ec67c7.slice/crio-22e227e9ed47ab66f360ef2289fbb452258231bb9de473fcf28c04c3c2393e98 WatchSource:0}: Error finding container 22e227e9ed47ab66f360ef2289fbb452258231bb9de473fcf28c04c3c2393e98: Status 404 returned error can't find the container with id 22e227e9ed47ab66f360ef2289fbb452258231bb9de473fcf28c04c3c2393e98
Sep 09 17:44:05.523879 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.523853    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"]
Sep 09 17:44:05.525327 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:44:05.525300    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af953a9_987d_4f15_b265_0c66b6fc5085.slice/crio-b92c276e06b22772d02863a77160888aef9ab40e7b51be100f6f6bf73c9634e0 WatchSource:0}: Error finding container b92c276e06b22772d02863a77160888aef9ab40e7b51be100f6f6bf73c9634e0: Status 404 returned error can't find the container with id b92c276e06b22772d02863a77160888aef9ab40e7b51be100f6f6bf73c9634e0
Sep 09 17:44:05.573932 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.573895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h" event={"ID":"172810f8-62cb-4db5-842b-b56547ec67c7","Type":"ContainerStarted","Data":"95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105"}
Sep 09 17:44:05.574131 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.573975    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h" event={"ID":"172810f8-62cb-4db5-842b-b56547ec67c7","Type":"ContainerStarted","Data":"22e227e9ed47ab66f360ef2289fbb452258231bb9de473fcf28c04c3c2393e98"}
Sep 09 17:44:05.574131 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.574014    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:05.575554 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.575530    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x" event={"ID":"8af953a9-987d-4f15-b265-0c66b6fc5085","Type":"ContainerStarted","Data":"b92c276e06b22772d02863a77160888aef9ab40e7b51be100f6f6bf73c9634e0"}
Sep 09 17:44:05.591481 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:05.591427    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h" podStartSLOduration=1.591409184 podStartE2EDuration="1.591409184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:44:05.589296658 +0000 UTC m=+1621.217449363" watchObservedRunningTime="2026-09-09 17:44:05.591409184 +0000 UTC m=+1621.219561886"
Sep 09 17:44:06.587330 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:06.587289    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x" event={"ID":"8af953a9-987d-4f15-b265-0c66b6fc5085","Type":"ContainerStarted","Data":"fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba"}
Sep 09 17:44:06.587745 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:06.587358    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:06.604831 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:06.604779    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x" podStartSLOduration=2.604762424 podStartE2EDuration="2.604762424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:44:06.603348594 +0000 UTC m=+1622.231501297" watchObservedRunningTime="2026-09-09 17:44:06.604762424 +0000 UTC m=+1622.232915168"
Sep 09 17:44:16.589538 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:16.589504    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:44:17.595059 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:17.595022    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:44:31.073352 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.073312    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"]
Sep 09 17:44:31.077036 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.077011    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"
Sep 09 17:44:31.080041 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.080016    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:44:31.084618 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.084586    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"]
Sep 09 17:44:31.271188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.271146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1a6c02-e126-45bd-805b-75fa636e8855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf91eb-6x288\" (UID: \"8d1a6c02-e126-45bd-805b-75fa636e8855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"
Sep 09 17:44:31.372337 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.372251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1a6c02-e126-45bd-805b-75fa636e8855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf91eb-6x288\" (UID: \"8d1a6c02-e126-45bd-805b-75fa636e8855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"
Sep 09 17:44:31.372840 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.372821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1a6c02-e126-45bd-805b-75fa636e8855-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf91eb-6x288\" (UID: \"8d1a6c02-e126-45bd-805b-75fa636e8855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"
Sep 09 17:44:31.391216 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.391195    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"
Sep 09 17:44:31.677049 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.677024    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"]
Sep 09 17:44:31.679053 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:44:31.679018    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1a6c02_e126_45bd_805b_75fa636e8855.slice/crio-91d50793ace6c6274fa8104a6c550d3d569a51a84da436855cec5614158b9458 WatchSource:0}: Error finding container 91d50793ace6c6274fa8104a6c550d3d569a51a84da436855cec5614158b9458: Status 404 returned error can't find the container with id 91d50793ace6c6274fa8104a6c550d3d569a51a84da436855cec5614158b9458
Sep 09 17:44:31.734833 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:31.734799    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288" event={"ID":"8d1a6c02-e126-45bd-805b-75fa636e8855","Type":"ContainerStarted","Data":"91d50793ace6c6274fa8104a6c550d3d569a51a84da436855cec5614158b9458"}
Sep 09 17:44:32.741770 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:32.741734    2558 generic.go:356] "Generic (PLEG): container finished" podID="8d1a6c02-e126-45bd-805b-75fa636e8855" containerID="4a477dc38001bbe20f9faf8d7cc06b9b7301a9bde35daba41146d746ce8cbcdb" exitCode=0
Sep 09 17:44:32.742196 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:32.741775    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288" event={"ID":"8d1a6c02-e126-45bd-805b-75fa636e8855","Type":"ContainerDied","Data":"4a477dc38001bbe20f9faf8d7cc06b9b7301a9bde35daba41146d746ce8cbcdb"}
Sep 09 17:44:33.891408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:33.891381    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"
Sep 09 17:44:33.892655 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:33.892637    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1a6c02-e126-45bd-805b-75fa636e8855-maas-api-service-ca\") pod \"8d1a6c02-e126-45bd-805b-75fa636e8855\" (UID: \"8d1a6c02-e126-45bd-805b-75fa636e8855\") "
Sep 09 17:44:33.892987 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:33.892941    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1a6c02-e126-45bd-805b-75fa636e8855-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d1a6c02-e126-45bd-805b-75fa636e8855" (UID: "8d1a6c02-e126-45bd-805b-75fa636e8855"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:44:33.993639 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:33.993600    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1a6c02-e126-45bd-805b-75fa636e8855-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:44:34.760040 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:34.760009    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"
Sep 09 17:44:34.760040 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:34.760019    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288" event={"ID":"8d1a6c02-e126-45bd-805b-75fa636e8855","Type":"ContainerDied","Data":"91d50793ace6c6274fa8104a6c550d3d569a51a84da436855cec5614158b9458"}
Sep 09 17:44:34.760280 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:34.760109    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d50793ace6c6274fa8104a6c550d3d569a51a84da436855cec5614158b9458"
Sep 09 17:44:46.541214 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.541180    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"]
Sep 09 17:44:46.541785 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.541427    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g" podUID="9212d022-7d83-4d21-be11-4b43e0d0aaf9" containerName="payload-pre-processing" containerID="cri-o://7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c" gracePeriod=30
Sep 09 17:44:46.693533 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.693493    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"]
Sep 09 17:44:46.696775 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.696748    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf91eb-6x288"]
Sep 09 17:44:46.791909 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.791848    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:44:46.817826 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.817791    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1a6c02-e126-45bd-805b-75fa636e8855" path="/var/lib/kubelet/pods/8d1a6c02-e126-45bd-805b-75fa636e8855/volumes"
Sep 09 17:44:46.829324 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.829296    2558 generic.go:356] "Generic (PLEG): container finished" podID="9212d022-7d83-4d21-be11-4b43e0d0aaf9" containerID="7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c" exitCode=0
Sep 09 17:44:46.829483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.829342    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"
Sep 09 17:44:46.829483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.829379    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g" event={"ID":"9212d022-7d83-4d21-be11-4b43e0d0aaf9","Type":"ContainerDied","Data":"7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c"}
Sep 09 17:44:46.829483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.829434    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g" event={"ID":"9212d022-7d83-4d21-be11-4b43e0d0aaf9","Type":"ContainerDied","Data":"dc4faaf2f630cd9c5665d4a98e5de8f45a8536d449bf45bea91dc06f96690c4e"}
Sep 09 17:44:46.829483 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.829462    2558 scope.go:117] "RemoveContainer" containerID="7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c"
Sep 09 17:44:46.839110 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.839071    2558 scope.go:117] "RemoveContainer" containerID="7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c"
Sep 09 17:44:46.839435 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:44:46.839415    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c\": container with ID starting with 7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c not found: ID does not exist" containerID="7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c"
Sep 09 17:44:46.839494 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.839443    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c"} err="failed to get container status \"7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c\": rpc error: code = NotFound desc = could not find container \"7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c\": container with ID starting with 7f09b4530f5509d5cb88c9c7be084081bf109b0ce73435f886860b902284c08c not found: ID does not exist"
Sep 09 17:44:46.888897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.888861    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9212d022-7d83-4d21-be11-4b43e0d0aaf9-plugins-config-volume\") pod \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") "
Sep 09 17:44:46.889096 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.888998    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx9f9\" (UniqueName: \"kubernetes.io/projected/9212d022-7d83-4d21-be11-4b43e0d0aaf9-kube-api-access-dx9f9\") pod \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\" (UID: \"9212d022-7d83-4d21-be11-4b43e0d0aaf9\") "
Sep 09 17:44:46.889308 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.889284    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9212d022-7d83-4d21-be11-4b43e0d0aaf9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9212d022-7d83-4d21-be11-4b43e0d0aaf9" (UID: "9212d022-7d83-4d21-be11-4b43e0d0aaf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:44:46.890985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.890966    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9212d022-7d83-4d21-be11-4b43e0d0aaf9-kube-api-access-dx9f9" (OuterVolumeSpecName: "kube-api-access-dx9f9") pod "9212d022-7d83-4d21-be11-4b43e0d0aaf9" (UID: "9212d022-7d83-4d21-be11-4b43e0d0aaf9"). InnerVolumeSpecName "kube-api-access-dx9f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:44:46.990527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.990488    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx9f9\" (UniqueName: \"kubernetes.io/projected/9212d022-7d83-4d21-be11-4b43e0d0aaf9-kube-api-access-dx9f9\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:44:46.990527 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:46.990519    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9212d022-7d83-4d21-be11-4b43e0d0aaf9-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:44:47.151572 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:47.151540    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"]
Sep 09 17:44:47.153558 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:47.153535    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf91eb-5d8cfc4798-8tq8g"]
Sep 09 17:44:48.817013 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:48.816979    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9212d022-7d83-4d21-be11-4b43e0d0aaf9" path="/var/lib/kubelet/pods/9212d022-7d83-4d21-be11-4b43e0d0aaf9/volumes"
Sep 09 17:44:53.449005 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.448949    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"]
Sep 09 17:44:53.449455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.449310    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1a6c02-e126-45bd-805b-75fa636e8855" containerName="revoke-keys"
Sep 09 17:44:53.449455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.449320    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1a6c02-e126-45bd-805b-75fa636e8855" containerName="revoke-keys"
Sep 09 17:44:53.449455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.449340    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9212d022-7d83-4d21-be11-4b43e0d0aaf9" containerName="payload-pre-processing"
Sep 09 17:44:53.449455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.449345    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9212d022-7d83-4d21-be11-4b43e0d0aaf9" containerName="payload-pre-processing"
Sep 09 17:44:53.449455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.449395    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1a6c02-e126-45bd-805b-75fa636e8855" containerName="revoke-keys"
Sep 09 17:44:53.449455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.449406    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9212d022-7d83-4d21-be11-4b43e0d0aaf9" containerName="payload-pre-processing"
Sep 09 17:44:53.453930 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.453913    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"
Sep 09 17:44:53.456973 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.456924    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:44:53.459178 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.459151    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"]
Sep 09 17:44:53.549662 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.549628    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5d323-837e-4bbd-9131-81f01ba6accb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf\" (UID: \"16d5d323-837e-4bbd-9131-81f01ba6accb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"
Sep 09 17:44:53.650360 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.650322    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5d323-837e-4bbd-9131-81f01ba6accb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf\" (UID: \"16d5d323-837e-4bbd-9131-81f01ba6accb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"
Sep 09 17:44:53.650949 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.650925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5d323-837e-4bbd-9131-81f01ba6accb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf\" (UID: \"16d5d323-837e-4bbd-9131-81f01ba6accb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"
Sep 09 17:44:53.765800 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:53.765712    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"
Sep 09 17:44:54.037901 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:54.037871    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"]
Sep 09 17:44:54.038892 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:44:54.038857    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d5d323_837e_4bbd_9131_81f01ba6accb.slice/crio-fad2b50c70af4f9a1e320ef9b4bc417510efbb24b498d9d60f40e20f32d221e6 WatchSource:0}: Error finding container fad2b50c70af4f9a1e320ef9b4bc417510efbb24b498d9d60f40e20f32d221e6: Status 404 returned error can't find the container with id fad2b50c70af4f9a1e320ef9b4bc417510efbb24b498d9d60f40e20f32d221e6
Sep 09 17:44:54.874963 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:54.874898    2558 generic.go:356] "Generic (PLEG): container finished" podID="16d5d323-837e-4bbd-9131-81f01ba6accb" containerID="1ad4efc9732e5cc5e5dd244508a241447f68088b328cda5a9124baeedc0ec4b9" exitCode=0
Sep 09 17:44:54.875376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:54.874985    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf" event={"ID":"16d5d323-837e-4bbd-9131-81f01ba6accb","Type":"ContainerDied","Data":"1ad4efc9732e5cc5e5dd244508a241447f68088b328cda5a9124baeedc0ec4b9"}
Sep 09 17:44:54.875376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:54.875054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf" event={"ID":"16d5d323-837e-4bbd-9131-81f01ba6accb","Type":"ContainerStarted","Data":"fad2b50c70af4f9a1e320ef9b4bc417510efbb24b498d9d60f40e20f32d221e6"}
Sep 09 17:44:56.011255 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:56.011229    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"
Sep 09 17:44:56.070462 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:56.070423    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5d323-837e-4bbd-9131-81f01ba6accb-maas-api-service-ca\") pod \"16d5d323-837e-4bbd-9131-81f01ba6accb\" (UID: \"16d5d323-837e-4bbd-9131-81f01ba6accb\") "
Sep 09 17:44:56.070831 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:56.070805    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d5d323-837e-4bbd-9131-81f01ba6accb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16d5d323-837e-4bbd-9131-81f01ba6accb" (UID: "16d5d323-837e-4bbd-9131-81f01ba6accb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:44:56.171132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:56.171018    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5d323-837e-4bbd-9131-81f01ba6accb-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:44:56.887266 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:56.887239    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"
Sep 09 17:44:56.887432 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:56.887237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf" event={"ID":"16d5d323-837e-4bbd-9131-81f01ba6accb","Type":"ContainerDied","Data":"fad2b50c70af4f9a1e320ef9b4bc417510efbb24b498d9d60f40e20f32d221e6"}
Sep 09 17:44:56.887432 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:44:56.887379    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad2b50c70af4f9a1e320ef9b4bc417510efbb24b498d9d60f40e20f32d221e6"
Sep 09 17:45:00.139744 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.139696    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"]
Sep 09 17:45:00.140371 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.140348    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d5d323-837e-4bbd-9131-81f01ba6accb" containerName="revoke-keys"
Sep 09 17:45:00.140417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.140375    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d5d323-837e-4bbd-9131-81f01ba6accb" containerName="revoke-keys"
Sep 09 17:45:00.140471 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.140460    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d5d323-837e-4bbd-9131-81f01ba6accb" containerName="revoke-keys"
Sep 09 17:45:00.144109 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.144083    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"]
Sep 09 17:45:00.144252 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.144195    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"
Sep 09 17:45:00.146643 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.146622    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:45:00.147353 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.147338    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"
Sep 09 17:45:00.153025 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.153003    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"]
Sep 09 17:45:00.156494 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.156471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"]
Sep 09 17:45:00.160057 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.160035    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"
Sep 09 17:45:00.167619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.167596    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"
Sep 09 17:45:00.449278 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.449249    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"]
Sep 09 17:45:00.450655 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:45:00.450629    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3256496a_2575_4077_866e_37935bba9d68.slice/crio-886f07d21386c44306301eda1ab2a1ef09047f64d2302083f669bf9353fc3557 WatchSource:0}: Error finding container 886f07d21386c44306301eda1ab2a1ef09047f64d2302083f669bf9353fc3557: Status 404 returned error can't find the container with id 886f07d21386c44306301eda1ab2a1ef09047f64d2302083f669bf9353fc3557
Sep 09 17:45:00.454880 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.454859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"]
Sep 09 17:45:00.456550 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:45:00.456518    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfce70f4_a9a4_4aa5_829b_79e0390d17c2.slice/crio-94e655a3e521c5c3487eca13d5c24ee13e4f73c0e9b6a6da304e13818340992f WatchSource:0}: Error finding container 94e655a3e521c5c3487eca13d5c24ee13e4f73c0e9b6a6da304e13818340992f: Status 404 returned error can't find the container with id 94e655a3e521c5c3487eca13d5c24ee13e4f73c0e9b6a6da304e13818340992f
Sep 09 17:45:00.910829 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.910792    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m" event={"ID":"cfce70f4-a9a4-4aa5-829b-79e0390d17c2","Type":"ContainerStarted","Data":"02480ed8a5500c4696a641c040665ebdd9210916fce44fc7dd698e76af955db6"}
Sep 09 17:45:00.911058 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.910848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m" event={"ID":"cfce70f4-a9a4-4aa5-829b-79e0390d17c2","Type":"ContainerStarted","Data":"94e655a3e521c5c3487eca13d5c24ee13e4f73c0e9b6a6da304e13818340992f"}
Sep 09 17:45:00.912550 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.912527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww" event={"ID":"3256496a-2575-4077-866e-37935bba9d68","Type":"ContainerStarted","Data":"f5756752c5078711f9ea8a69ce905949642698ba6d8d34d59565f98f545454be"}
Sep 09 17:45:00.912674 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.912572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww" event={"ID":"3256496a-2575-4077-866e-37935bba9d68","Type":"ContainerStarted","Data":"886f07d21386c44306301eda1ab2a1ef09047f64d2302083f669bf9353fc3557"}
Sep 09 17:45:00.926450 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.926404    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m" podStartSLOduration=0.926381191 podStartE2EDuration="926.381191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:45:00.924769387 +0000 UTC m=+1676.552922094" watchObservedRunningTime="2026-09-09 17:45:00.926381191 +0000 UTC m=+1676.554533895"
Sep 09 17:45:00.938856 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:00.938803    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww" podStartSLOduration=0.938785936 podStartE2EDuration="938.785936ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:45:00.937506298 +0000 UTC m=+1676.565658999" watchObservedRunningTime="2026-09-09 17:45:00.938785936 +0000 UTC m=+1676.566938631"
Sep 09 17:45:01.920216 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:01.920127    2558 generic.go:356] "Generic (PLEG): container finished" podID="cfce70f4-a9a4-4aa5-829b-79e0390d17c2" containerID="02480ed8a5500c4696a641c040665ebdd9210916fce44fc7dd698e76af955db6" exitCode=0
Sep 09 17:45:01.920672 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:01.920210    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m" event={"ID":"cfce70f4-a9a4-4aa5-829b-79e0390d17c2","Type":"ContainerDied","Data":"02480ed8a5500c4696a641c040665ebdd9210916fce44fc7dd698e76af955db6"}
Sep 09 17:45:01.921816 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:01.921794    2558 generic.go:356] "Generic (PLEG): container finished" podID="3256496a-2575-4077-866e-37935bba9d68" containerID="f5756752c5078711f9ea8a69ce905949642698ba6d8d34d59565f98f545454be" exitCode=0
Sep 09 17:45:01.921922 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:01.921879    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww" event={"ID":"3256496a-2575-4077-866e-37935bba9d68","Type":"ContainerDied","Data":"f5756752c5078711f9ea8a69ce905949642698ba6d8d34d59565f98f545454be"}
Sep 09 17:45:03.082692 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.082669    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"
Sep 09 17:45:03.085387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.085366    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"
Sep 09 17:45:03.935095 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.935045    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"
Sep 09 17:45:03.935297 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.935097    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww" event={"ID":"3256496a-2575-4077-866e-37935bba9d68","Type":"ContainerDied","Data":"886f07d21386c44306301eda1ab2a1ef09047f64d2302083f669bf9353fc3557"}
Sep 09 17:45:03.935297 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.935146    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886f07d21386c44306301eda1ab2a1ef09047f64d2302083f669bf9353fc3557"
Sep 09 17:45:03.936439 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.936413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m" event={"ID":"cfce70f4-a9a4-4aa5-829b-79e0390d17c2","Type":"ContainerDied","Data":"94e655a3e521c5c3487eca13d5c24ee13e4f73c0e9b6a6da304e13818340992f"}
Sep 09 17:45:03.936439 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.936431    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"
Sep 09 17:45:03.936596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:03.936458    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e655a3e521c5c3487eca13d5c24ee13e4f73c0e9b6a6da304e13818340992f"
Sep 09 17:45:05.049034 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:05.045772    2558 scope.go:117] "RemoveContainer" containerID="a5b1bec0ecb0d5725110325268ebbdb6dec31174acd7dbe87ab42803be5bf7d6"
Sep 09 17:45:05.057568 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:05.057548    2558 scope.go:117] "RemoveContainer" containerID="7447f726cbb706de21b64527aa3344fbddadc26c0edc262d1d3f5012fefa8154"
Sep 09 17:45:05.065444 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:05.065425    2558 scope.go:117] "RemoveContainer" containerID="3d1d318b9e4505fba2990ea52938b64d290c508ce07ed380b48d43105189e6fd"
Sep 09 17:45:05.073120 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:05.073104    2558 scope.go:117] "RemoveContainer" containerID="2caf5975a54f076723f1e9302708c51449de5c24e9e776223488382937e9328b"
Sep 09 17:45:09.018503 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.018473    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"]
Sep 09 17:45:09.020713 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.020686    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ecf58e-29816265-fm6ww"]
Sep 09 17:45:09.049300 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.049262    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"]
Sep 09 17:45:09.049556 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.049527    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x" podUID="8af953a9-987d-4f15-b265-0c66b6fc5085" containerName="payload-processing" containerID="cri-o://fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba" gracePeriod=30
Sep 09 17:45:09.054271 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.054242    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"]
Sep 09 17:45:09.054628 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.054597    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h" podUID="172810f8-62cb-4db5-842b-b56547ec67c7" containerName="payload-pre-processing" containerID="cri-o://95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105" gracePeriod=30
Sep 09 17:45:09.192715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.192680    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"]
Sep 09 17:45:09.194811 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.194781    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecf58e-sdwsf"]
Sep 09 17:45:09.328386 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.328361    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:45:09.330901 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.330881    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:45:09.385486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.385449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hcsh\" (UniqueName: \"kubernetes.io/projected/172810f8-62cb-4db5-842b-b56547ec67c7-kube-api-access-6hcsh\") pod \"172810f8-62cb-4db5-842b-b56547ec67c7\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") "
Sep 09 17:45:09.385666 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.385577    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172810f8-62cb-4db5-842b-b56547ec67c7-plugins-config-volume\") pod \"172810f8-62cb-4db5-842b-b56547ec67c7\" (UID: \"172810f8-62cb-4db5-842b-b56547ec67c7\") "
Sep 09 17:45:09.385666 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.385605    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af953a9-987d-4f15-b265-0c66b6fc5085-plugins-config-volume\") pod \"8af953a9-987d-4f15-b265-0c66b6fc5085\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") "
Sep 09 17:45:09.385666 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.385637    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtc8k\" (UniqueName: \"kubernetes.io/projected/8af953a9-987d-4f15-b265-0c66b6fc5085-kube-api-access-qtc8k\") pod \"8af953a9-987d-4f15-b265-0c66b6fc5085\" (UID: \"8af953a9-987d-4f15-b265-0c66b6fc5085\") "
Sep 09 17:45:09.385931 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.385909    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172810f8-62cb-4db5-842b-b56547ec67c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "172810f8-62cb-4db5-842b-b56547ec67c7" (UID: "172810f8-62cb-4db5-842b-b56547ec67c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:09.386020 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.385995    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af953a9-987d-4f15-b265-0c66b6fc5085-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8af953a9-987d-4f15-b265-0c66b6fc5085" (UID: "8af953a9-987d-4f15-b265-0c66b6fc5085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:09.387601 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.387573    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172810f8-62cb-4db5-842b-b56547ec67c7-kube-api-access-6hcsh" (OuterVolumeSpecName: "kube-api-access-6hcsh") pod "172810f8-62cb-4db5-842b-b56547ec67c7" (UID: "172810f8-62cb-4db5-842b-b56547ec67c7"). InnerVolumeSpecName "kube-api-access-6hcsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:45:09.387712 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.387633    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af953a9-987d-4f15-b265-0c66b6fc5085-kube-api-access-qtc8k" (OuterVolumeSpecName: "kube-api-access-qtc8k") pod "8af953a9-987d-4f15-b265-0c66b6fc5085" (UID: "8af953a9-987d-4f15-b265-0c66b6fc5085"). InnerVolumeSpecName "kube-api-access-qtc8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:45:09.486619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.486575    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hcsh\" (UniqueName: \"kubernetes.io/projected/172810f8-62cb-4db5-842b-b56547ec67c7-kube-api-access-6hcsh\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:09.486619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.486610    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172810f8-62cb-4db5-842b-b56547ec67c7-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:09.486619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.486622    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af953a9-987d-4f15-b265-0c66b6fc5085-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:09.486619 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.486630    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtc8k\" (UniqueName: \"kubernetes.io/projected/8af953a9-987d-4f15-b265-0c66b6fc5085-kube-api-access-qtc8k\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:09.969825 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.969790    2558 generic.go:356] "Generic (PLEG): container finished" podID="172810f8-62cb-4db5-842b-b56547ec67c7" containerID="95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105" exitCode=0
Sep 09 17:45:09.970036 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.969831    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"
Sep 09 17:45:09.970036 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.969877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h" event={"ID":"172810f8-62cb-4db5-842b-b56547ec67c7","Type":"ContainerDied","Data":"95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105"}
Sep 09 17:45:09.970036 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.969939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h" event={"ID":"172810f8-62cb-4db5-842b-b56547ec67c7","Type":"ContainerDied","Data":"22e227e9ed47ab66f360ef2289fbb452258231bb9de473fcf28c04c3c2393e98"}
Sep 09 17:45:09.970036 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.969985    2558 scope.go:117] "RemoveContainer" containerID="95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105"
Sep 09 17:45:09.971312 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.971289    2558 generic.go:356] "Generic (PLEG): container finished" podID="8af953a9-987d-4f15-b265-0c66b6fc5085" containerID="fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba" exitCode=0
Sep 09 17:45:09.971419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.971332    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"
Sep 09 17:45:09.971419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.971387    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x" event={"ID":"8af953a9-987d-4f15-b265-0c66b6fc5085","Type":"ContainerDied","Data":"fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba"}
Sep 09 17:45:09.971532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.971427    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x" event={"ID":"8af953a9-987d-4f15-b265-0c66b6fc5085","Type":"ContainerDied","Data":"b92c276e06b22772d02863a77160888aef9ab40e7b51be100f6f6bf73c9634e0"}
Sep 09 17:45:09.979445 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.979431    2558 scope.go:117] "RemoveContainer" containerID="95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105"
Sep 09 17:45:09.979737 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:45:09.979717    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105\": container with ID starting with 95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105 not found: ID does not exist" containerID="95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105"
Sep 09 17:45:09.979814 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.979751    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105"} err="failed to get container status \"95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105\": rpc error: code = NotFound desc = could not find container \"95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105\": container with ID starting with 95c059fe989e7f2a58f700ceb036b4f05196a7fdb88306f5bed2ba14ce199105 not found: ID does not exist"
Sep 09 17:45:09.979814 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.979776    2558 scope.go:117] "RemoveContainer" containerID="fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba"
Sep 09 17:45:09.987982 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.987964    2558 scope.go:117] "RemoveContainer" containerID="fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba"
Sep 09 17:45:09.988250 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:45:09.988229    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba\": container with ID starting with fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba not found: ID does not exist" containerID="fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba"
Sep 09 17:45:09.988314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.988257    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba"} err="failed to get container status \"fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba\": rpc error: code = NotFound desc = could not find container \"fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba\": container with ID starting with fd3b6ce57fa231a3765b0657598357ba8ce43a9f877d45eb5ece11f34c1a31ba not found: ID does not exist"
Sep 09 17:45:09.994841 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:09.994812    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"]
Sep 09 17:45:10.000317 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:10.000286    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecf58e-5484959746-qj66h"]
Sep 09 17:45:10.010820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:10.010783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"]
Sep 09 17:45:10.013710 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:10.013687    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecf58e-59c5779ccc-h785x"]
Sep 09 17:45:10.816869 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:10.816835    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16d5d323-837e-4bbd-9131-81f01ba6accb" path="/var/lib/kubelet/pods/16d5d323-837e-4bbd-9131-81f01ba6accb/volumes"
Sep 09 17:45:10.817364 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:10.817212    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172810f8-62cb-4db5-842b-b56547ec67c7" path="/var/lib/kubelet/pods/172810f8-62cb-4db5-842b-b56547ec67c7/volumes"
Sep 09 17:45:10.817451 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:10.817437    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3256496a-2575-4077-866e-37935bba9d68" path="/var/lib/kubelet/pods/3256496a-2575-4077-866e-37935bba9d68/volumes"
Sep 09 17:45:10.817713 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:10.817702    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8af953a9-987d-4f15-b265-0c66b6fc5085" path="/var/lib/kubelet/pods/8af953a9-987d-4f15-b265-0c66b6fc5085/volumes"
Sep 09 17:45:15.870041 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870006    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"]
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870343    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172810f8-62cb-4db5-842b-b56547ec67c7" containerName="payload-pre-processing"
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870355    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="172810f8-62cb-4db5-842b-b56547ec67c7" containerName="payload-pre-processing"
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870364    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3256496a-2575-4077-866e-37935bba9d68" containerName="cleanup"
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870369    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3256496a-2575-4077-866e-37935bba9d68" containerName="cleanup"
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870383    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfce70f4-a9a4-4aa5-829b-79e0390d17c2" containerName="cleanup"
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870389    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfce70f4-a9a4-4aa5-829b-79e0390d17c2" containerName="cleanup"
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870396    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af953a9-987d-4f15-b265-0c66b6fc5085" containerName="payload-processing"
Sep 09 17:45:15.870401 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870401    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af953a9-987d-4f15-b265-0c66b6fc5085" containerName="payload-processing"
Sep 09 17:45:15.870647 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870451    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfce70f4-a9a4-4aa5-829b-79e0390d17c2" containerName="cleanup"
Sep 09 17:45:15.870647 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870458    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af953a9-987d-4f15-b265-0c66b6fc5085" containerName="payload-processing"
Sep 09 17:45:15.870647 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870468    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="172810f8-62cb-4db5-842b-b56547ec67c7" containerName="payload-pre-processing"
Sep 09 17:45:15.870647 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.870473    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3256496a-2575-4077-866e-37935bba9d68" containerName="cleanup"
Sep 09 17:45:15.873455 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.873439    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"
Sep 09 17:45:15.876188 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.876162    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:45:15.881588 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.881565    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"]
Sep 09 17:45:15.940809 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:15.940769    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5cf549a-2fac-456f-a97b-d3454b2d1a05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp\" (UID: \"e5cf549a-2fac-456f-a97b-d3454b2d1a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"
Sep 09 17:45:16.041374 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:16.041339    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5cf549a-2fac-456f-a97b-d3454b2d1a05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp\" (UID: \"e5cf549a-2fac-456f-a97b-d3454b2d1a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"
Sep 09 17:45:16.041925 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:16.041903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5cf549a-2fac-456f-a97b-d3454b2d1a05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp\" (UID: \"e5cf549a-2fac-456f-a97b-d3454b2d1a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"
Sep 09 17:45:16.185737 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:16.185702    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"
Sep 09 17:45:16.455830 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:16.455805    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"]
Sep 09 17:45:16.457199 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:45:16.457171    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cf549a_2fac_456f_a97b_d3454b2d1a05.slice/crio-d1984b08a0ba511c0407b84f501725d4bc349f4f3a9cf21daf4d64f74ecdde4f WatchSource:0}: Error finding container d1984b08a0ba511c0407b84f501725d4bc349f4f3a9cf21daf4d64f74ecdde4f: Status 404 returned error can't find the container with id d1984b08a0ba511c0407b84f501725d4bc349f4f3a9cf21daf4d64f74ecdde4f
Sep 09 17:45:17.012888 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:17.012795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp" event={"ID":"e5cf549a-2fac-456f-a97b-d3454b2d1a05","Type":"ContainerStarted","Data":"17c7c54b45de10e2c8d16ee29dd2854394894560655f756679fa96248b6164d2"}
Sep 09 17:45:17.012888 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:17.012850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp" event={"ID":"e5cf549a-2fac-456f-a97b-d3454b2d1a05","Type":"ContainerStarted","Data":"d1984b08a0ba511c0407b84f501725d4bc349f4f3a9cf21daf4d64f74ecdde4f"}
Sep 09 17:45:17.027629 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:17.027580    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp" podStartSLOduration=2.027566013 podStartE2EDuration="2.027566013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:45:17.026836802 +0000 UTC m=+1692.654989505" watchObservedRunningTime="2026-09-09 17:45:17.027566013 +0000 UTC m=+1692.655718725"
Sep 09 17:45:18.019838 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:18.019801    2558 generic.go:356] "Generic (PLEG): container finished" podID="e5cf549a-2fac-456f-a97b-d3454b2d1a05" containerID="17c7c54b45de10e2c8d16ee29dd2854394894560655f756679fa96248b6164d2" exitCode=0
Sep 09 17:45:18.020286 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:18.019881    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp" event={"ID":"e5cf549a-2fac-456f-a97b-d3454b2d1a05","Type":"ContainerDied","Data":"17c7c54b45de10e2c8d16ee29dd2854394894560655f756679fa96248b6164d2"}
Sep 09 17:45:19.148036 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:19.148011    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"
Sep 09 17:45:19.269855 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:19.269817    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5cf549a-2fac-456f-a97b-d3454b2d1a05-maas-api-service-ca\") pod \"e5cf549a-2fac-456f-a97b-d3454b2d1a05\" (UID: \"e5cf549a-2fac-456f-a97b-d3454b2d1a05\") "
Sep 09 17:45:19.270213 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:19.270188    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cf549a-2fac-456f-a97b-d3454b2d1a05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5cf549a-2fac-456f-a97b-d3454b2d1a05" (UID: "e5cf549a-2fac-456f-a97b-d3454b2d1a05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:19.371433 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:19.371346    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5cf549a-2fac-456f-a97b-d3454b2d1a05-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:20.031901 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:20.031865    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"
Sep 09 17:45:20.031901 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:20.031892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp" event={"ID":"e5cf549a-2fac-456f-a97b-d3454b2d1a05","Type":"ContainerDied","Data":"d1984b08a0ba511c0407b84f501725d4bc349f4f3a9cf21daf4d64f74ecdde4f"}
Sep 09 17:45:20.032139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:20.031939    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1984b08a0ba511c0407b84f501725d4bc349f4f3a9cf21daf4d64f74ecdde4f"
Sep 09 17:45:26.316540 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.316503    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"]
Sep 09 17:45:26.318516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.318491    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b44e3-29816265-csv9m"]
Sep 09 17:45:26.340846 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.340808    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"]
Sep 09 17:45:26.341113 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.341084    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt" podUID="d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" containerName="payload-processing" containerID="cri-o://1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90" gracePeriod=30
Sep 09 17:45:26.350492 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.350459    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"]
Sep 09 17:45:26.350717 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.350694    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg" podUID="59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" containerName="payload-pre-processing" containerID="cri-o://4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb" gracePeriod=30
Sep 09 17:45:26.498729 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.498674    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"]
Sep 09 17:45:26.500690 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.500666    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b44e3-w44rp"]
Sep 09 17:45:26.612096 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.612069    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:45:26.614816 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.614798    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:45:26.737981 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.737919    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-plugins-config-volume\") pod \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") "
Sep 09 17:45:26.738172 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.737998    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-plugins-config-volume\") pod \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") "
Sep 09 17:45:26.738172 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.738038    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45ctk\" (UniqueName: \"kubernetes.io/projected/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-kube-api-access-45ctk\") pod \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\" (UID: \"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae\") "
Sep 09 17:45:26.738172 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.738064    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flsgp\" (UniqueName: \"kubernetes.io/projected/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-kube-api-access-flsgp\") pod \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\" (UID: \"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b\") "
Sep 09 17:45:26.738417 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.738388    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" (UID: "d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:26.738470 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.738388    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" (UID: "59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:26.740286 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.740250    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-kube-api-access-45ctk" (OuterVolumeSpecName: "kube-api-access-45ctk") pod "59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" (UID: "59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae"). InnerVolumeSpecName "kube-api-access-45ctk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:45:26.740384 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.740292    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-kube-api-access-flsgp" (OuterVolumeSpecName: "kube-api-access-flsgp") pod "d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" (UID: "d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b"). InnerVolumeSpecName "kube-api-access-flsgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:45:26.816793 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.816756    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfce70f4-a9a4-4aa5-829b-79e0390d17c2" path="/var/lib/kubelet/pods/cfce70f4-a9a4-4aa5-829b-79e0390d17c2/volumes"
Sep 09 17:45:26.817085 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.817071    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5cf549a-2fac-456f-a97b-d3454b2d1a05" path="/var/lib/kubelet/pods/e5cf549a-2fac-456f-a97b-d3454b2d1a05/volumes"
Sep 09 17:45:26.839192 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.839160    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:26.839192 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.839189    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:26.839192 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.839198    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ctk\" (UniqueName: \"kubernetes.io/projected/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae-kube-api-access-45ctk\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:26.839411 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:26.839207    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flsgp\" (UniqueName: \"kubernetes.io/projected/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b-kube-api-access-flsgp\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:27.077715 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.077679    2558 generic.go:356] "Generic (PLEG): container finished" podID="59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" containerID="4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb" exitCode=0
Sep 09 17:45:27.077919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.077734    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"
Sep 09 17:45:27.077919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.077758    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg" event={"ID":"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae","Type":"ContainerDied","Data":"4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb"}
Sep 09 17:45:27.077919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.077812    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg" event={"ID":"59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae","Type":"ContainerDied","Data":"11c70169f080289cc0e38d231c69c5ec8c028efb0f815feafebc71261163a172"}
Sep 09 17:45:27.077919 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.077841    2558 scope.go:117] "RemoveContainer" containerID="4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb"
Sep 09 17:45:27.079302 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.079277    2558 generic.go:356] "Generic (PLEG): container finished" podID="d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" containerID="1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90" exitCode=0
Sep 09 17:45:27.079416 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.079320    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"
Sep 09 17:45:27.079416 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.079323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt" event={"ID":"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b","Type":"ContainerDied","Data":"1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90"}
Sep 09 17:45:27.079547 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.079430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt" event={"ID":"d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b","Type":"ContainerDied","Data":"629c47fee9fcd1a75ae85dd096afb283926e94def08c143a9ceba453af274ee8"}
Sep 09 17:45:27.087199 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.087181    2558 scope.go:117] "RemoveContainer" containerID="4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb"
Sep 09 17:45:27.087539 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:45:27.087516    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb\": container with ID starting with 4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb not found: ID does not exist" containerID="4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb"
Sep 09 17:45:27.087599 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.087546    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb"} err="failed to get container status \"4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb\": rpc error: code = NotFound desc = could not find container \"4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb\": container with ID starting with 4f7d7c8f637c2283737127af5099de5fe36c62aacee6600f01c2dc4f1772e5eb not found: ID does not exist"
Sep 09 17:45:27.087599 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.087563    2558 scope.go:117] "RemoveContainer" containerID="1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90"
Sep 09 17:45:27.096368 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.096339    2558 scope.go:117] "RemoveContainer" containerID="1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90"
Sep 09 17:45:27.096684 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:45:27.096666    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90\": container with ID starting with 1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90 not found: ID does not exist" containerID="1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90"
Sep 09 17:45:27.096748 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.096691    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90"} err="failed to get container status \"1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90\": rpc error: code = NotFound desc = could not find container \"1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90\": container with ID starting with 1fd2d17d25ee2284b44dcb91d6390db6242ed522dab1798118cfb14356a13a90 not found: ID does not exist"
Sep 09 17:45:27.097305 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.097286    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"]
Sep 09 17:45:27.101124 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.101047    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b44e3-79dc6bb6d6-q4qlg"]
Sep 09 17:45:27.111019 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.110992    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"]
Sep 09 17:45:27.115819 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:27.115788    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b44e3-8df899445-55kqt"]
Sep 09 17:45:28.816852 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:28.816806    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" path="/var/lib/kubelet/pods/59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae/volumes"
Sep 09 17:45:28.817366 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:28.817348    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" path="/var/lib/kubelet/pods/d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b/volumes"
Sep 09 17:45:33.456486 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.456442    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"]
Sep 09 17:45:33.456983 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.456908    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" containerName="payload-pre-processing"
Sep 09 17:45:33.456983 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.456927    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" containerName="payload-pre-processing"
Sep 09 17:45:33.456983 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.456968    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5cf549a-2fac-456f-a97b-d3454b2d1a05" containerName="revoke-keys"
Sep 09 17:45:33.456983 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.456977    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5cf549a-2fac-456f-a97b-d3454b2d1a05" containerName="revoke-keys"
Sep 09 17:45:33.457220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.456996    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" containerName="payload-processing"
Sep 09 17:45:33.457220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.457004    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" containerName="payload-processing"
Sep 09 17:45:33.457220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.457120    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d37f9c4b-9fc1-49eb-b8cc-cff52287ba1b" containerName="payload-processing"
Sep 09 17:45:33.457220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.457133    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="59df4e61-4e6d-4452-b9d8-f4dfbdcfb3ae" containerName="payload-pre-processing"
Sep 09 17:45:33.457220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.457144    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5cf549a-2fac-456f-a97b-d3454b2d1a05" containerName="revoke-keys"
Sep 09 17:45:33.461523 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.461501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"
Sep 09 17:45:33.464583 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.464516    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\""
Sep 09 17:45:33.466674 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.466647    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"]
Sep 09 17:45:33.596849 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.596803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ef4d6-026a-4b40-802c-5c64c5484f50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d\" (UID: \"750ef4d6-026a-4b40-802c-5c64c5484f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"
Sep 09 17:45:33.697553 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.697509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ef4d6-026a-4b40-802c-5c64c5484f50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d\" (UID: \"750ef4d6-026a-4b40-802c-5c64c5484f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"
Sep 09 17:45:33.698181 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.698161    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ef4d6-026a-4b40-802c-5c64c5484f50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d\" (UID: \"750ef4d6-026a-4b40-802c-5c64c5484f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"
Sep 09 17:45:33.772923 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:33.772827    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"
Sep 09 17:45:34.043757 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:34.043728    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"]
Sep 09 17:45:34.045664 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:45:34.045635    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750ef4d6_026a_4b40_802c_5c64c5484f50.slice/crio-9a02e2606af5ff8c456c58115139bd1fbfb1bc254afee778f93fe60c0053f9c1 WatchSource:0}: Error finding container 9a02e2606af5ff8c456c58115139bd1fbfb1bc254afee778f93fe60c0053f9c1: Status 404 returned error can't find the container with id 9a02e2606af5ff8c456c58115139bd1fbfb1bc254afee778f93fe60c0053f9c1
Sep 09 17:45:34.125376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:34.125333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d" event={"ID":"750ef4d6-026a-4b40-802c-5c64c5484f50","Type":"ContainerStarted","Data":"9a02e2606af5ff8c456c58115139bd1fbfb1bc254afee778f93fe60c0053f9c1"}
Sep 09 17:45:35.132766 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:35.132668    2558 generic.go:356] "Generic (PLEG): container finished" podID="750ef4d6-026a-4b40-802c-5c64c5484f50" containerID="a2b9eb0fd37ef6e20b739d1756cfe81c32a2db48f6459bd1ef8d0d23601aeff6" exitCode=0
Sep 09 17:45:35.132766 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:35.132719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d" event={"ID":"750ef4d6-026a-4b40-802c-5c64c5484f50","Type":"ContainerDied","Data":"a2b9eb0fd37ef6e20b739d1756cfe81c32a2db48f6459bd1ef8d0d23601aeff6"}
Sep 09 17:45:36.269866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:36.269842    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"
Sep 09 17:45:36.423227 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:36.423136    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ef4d6-026a-4b40-802c-5c64c5484f50-maas-api-service-ca\") pod \"750ef4d6-026a-4b40-802c-5c64c5484f50\" (UID: \"750ef4d6-026a-4b40-802c-5c64c5484f50\") "
Sep 09 17:45:36.423574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:36.423552    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750ef4d6-026a-4b40-802c-5c64c5484f50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "750ef4d6-026a-4b40-802c-5c64c5484f50" (UID: "750ef4d6-026a-4b40-802c-5c64c5484f50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:36.528980 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:36.525337    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ef4d6-026a-4b40-802c-5c64c5484f50-maas-api-service-ca\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:37.144734 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:37.144648    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"
Sep 09 17:45:37.144734 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:37.144690    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d" event={"ID":"750ef4d6-026a-4b40-802c-5c64c5484f50","Type":"ContainerDied","Data":"9a02e2606af5ff8c456c58115139bd1fbfb1bc254afee778f93fe60c0053f9c1"}
Sep 09 17:45:37.144937 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:37.144747    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a02e2606af5ff8c456c58115139bd1fbfb1bc254afee778f93fe60c0053f9c1"
Sep 09 17:45:43.970036 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:43.970001    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"]
Sep 09 17:45:43.970563 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:43.970211    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq" podUID="10e69cbb-2904-4ce0-a62b-ccac89a78922" containerName="payload-processing" containerID="cri-o://064083543b9c49d09880ecc57d263125ca3c2b2b43e509b308993f77d6816bac" gracePeriod=30
Sep 09 17:45:43.980067 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:43.980035    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"]
Sep 09 17:45:43.980302 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:43.980278    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c" podUID="4f6299fa-a3cb-49e4-84d2-448abadf0784" containerName="payload-pre-processing" containerID="cri-o://fbd727a0bab09ad757312cdfdcf2addc5c72e0f2cf9ed4f0e8fa42a3e3b9ad32" gracePeriod=30
Sep 09 17:45:44.125007 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.124934    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"]
Sep 09 17:45:44.126820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.126797    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-607751-mtp5d"]
Sep 09 17:45:44.187158 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.187110    2558 generic.go:356] "Generic (PLEG): container finished" podID="10e69cbb-2904-4ce0-a62b-ccac89a78922" containerID="064083543b9c49d09880ecc57d263125ca3c2b2b43e509b308993f77d6816bac" exitCode=0
Sep 09 17:45:44.187319 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.187183    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq" event={"ID":"10e69cbb-2904-4ce0-a62b-ccac89a78922","Type":"ContainerDied","Data":"064083543b9c49d09880ecc57d263125ca3c2b2b43e509b308993f77d6816bac"}
Sep 09 17:45:44.189059 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.189026    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f6299fa-a3cb-49e4-84d2-448abadf0784" containerID="fbd727a0bab09ad757312cdfdcf2addc5c72e0f2cf9ed4f0e8fa42a3e3b9ad32" exitCode=0
Sep 09 17:45:44.189184 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.189074    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c" event={"ID":"4f6299fa-a3cb-49e4-84d2-448abadf0784","Type":"ContainerDied","Data":"fbd727a0bab09ad757312cdfdcf2addc5c72e0f2cf9ed4f0e8fa42a3e3b9ad32"}
Sep 09 17:45:44.247859 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.247837    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:45:44.250725 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.250702    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:45:44.387012 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.386946    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6299fa-a3cb-49e4-84d2-448abadf0784-plugins-config-volume\") pod \"4f6299fa-a3cb-49e4-84d2-448abadf0784\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") "
Sep 09 17:45:44.387220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.387074    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e69cbb-2904-4ce0-a62b-ccac89a78922-plugins-config-volume\") pod \"10e69cbb-2904-4ce0-a62b-ccac89a78922\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") "
Sep 09 17:45:44.387220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.387119    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx67v\" (UniqueName: \"kubernetes.io/projected/10e69cbb-2904-4ce0-a62b-ccac89a78922-kube-api-access-lx67v\") pod \"10e69cbb-2904-4ce0-a62b-ccac89a78922\" (UID: \"10e69cbb-2904-4ce0-a62b-ccac89a78922\") "
Sep 09 17:45:44.387220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.387158    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkz5w\" (UniqueName: \"kubernetes.io/projected/4f6299fa-a3cb-49e4-84d2-448abadf0784-kube-api-access-pkz5w\") pod \"4f6299fa-a3cb-49e4-84d2-448abadf0784\" (UID: \"4f6299fa-a3cb-49e4-84d2-448abadf0784\") "
Sep 09 17:45:44.387381 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.387335    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6299fa-a3cb-49e4-84d2-448abadf0784-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f6299fa-a3cb-49e4-84d2-448abadf0784" (UID: "4f6299fa-a3cb-49e4-84d2-448abadf0784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:44.387422 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.387404    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6299fa-a3cb-49e4-84d2-448abadf0784-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:44.387468 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.387420    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e69cbb-2904-4ce0-a62b-ccac89a78922-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10e69cbb-2904-4ce0-a62b-ccac89a78922" (UID: "10e69cbb-2904-4ce0-a62b-ccac89a78922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 17:45:44.389277 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.389252    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6299fa-a3cb-49e4-84d2-448abadf0784-kube-api-access-pkz5w" (OuterVolumeSpecName: "kube-api-access-pkz5w") pod "4f6299fa-a3cb-49e4-84d2-448abadf0784" (UID: "4f6299fa-a3cb-49e4-84d2-448abadf0784"). InnerVolumeSpecName "kube-api-access-pkz5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:45:44.389335 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.389297    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e69cbb-2904-4ce0-a62b-ccac89a78922-kube-api-access-lx67v" (OuterVolumeSpecName: "kube-api-access-lx67v") pod "10e69cbb-2904-4ce0-a62b-ccac89a78922" (UID: "10e69cbb-2904-4ce0-a62b-ccac89a78922"). InnerVolumeSpecName "kube-api-access-lx67v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:45:44.488644 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.488557    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx67v\" (UniqueName: \"kubernetes.io/projected/10e69cbb-2904-4ce0-a62b-ccac89a78922-kube-api-access-lx67v\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:44.488644 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.488585    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkz5w\" (UniqueName: \"kubernetes.io/projected/4f6299fa-a3cb-49e4-84d2-448abadf0784-kube-api-access-pkz5w\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:44.488644 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.488594    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e69cbb-2904-4ce0-a62b-ccac89a78922-plugins-config-volume\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:44.816158 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:44.816075    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="750ef4d6-026a-4b40-802c-5c64c5484f50" path="/var/lib/kubelet/pods/750ef4d6-026a-4b40-802c-5c64c5484f50/volumes"
Sep 09 17:45:45.195833 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.195803    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"
Sep 09 17:45:45.196338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.195841    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c" event={"ID":"4f6299fa-a3cb-49e4-84d2-448abadf0784","Type":"ContainerDied","Data":"a2d1a8505685e3dbb2f4bf905fe0e9bfc90983be70ad103677c5c741e56bfc14"}
Sep 09 17:45:45.196338 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.195907    2558 scope.go:117] "RemoveContainer" containerID="fbd727a0bab09ad757312cdfdcf2addc5c72e0f2cf9ed4f0e8fa42a3e3b9ad32"
Sep 09 17:45:45.197408 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.197377    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq" event={"ID":"10e69cbb-2904-4ce0-a62b-ccac89a78922","Type":"ContainerDied","Data":"ec8fb963ee28ac114cfa4d947b1b1c26da52d42ec66d66d00d95ae8d4c9b4f78"}
Sep 09 17:45:45.197519 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.197392    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"
Sep 09 17:45:45.205814 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.205792    2558 scope.go:117] "RemoveContainer" containerID="064083543b9c49d09880ecc57d263125ca3c2b2b43e509b308993f77d6816bac"
Sep 09 17:45:45.215485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.215460    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"]
Sep 09 17:45:45.218727 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.218703    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-607751-7988fc4795-qsrsq"]
Sep 09 17:45:45.231604 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.231576    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"]
Sep 09 17:45:45.235228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:45.235205    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-607751-647dc67686-m4n2c"]
Sep 09 17:45:46.816382 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:46.816345    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10e69cbb-2904-4ce0-a62b-ccac89a78922" path="/var/lib/kubelet/pods/10e69cbb-2904-4ce0-a62b-ccac89a78922/volumes"
Sep 09 17:45:46.817038 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:46.817017    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6299fa-a3cb-49e4-84d2-448abadf0784" path="/var/lib/kubelet/pods/4f6299fa-a3cb-49e4-84d2-448abadf0784/volumes"
Sep 09 17:45:53.335460 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.335421    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-r87sq"]
Sep 09 17:45:53.335934 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.335648    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5474dd89c7-r87sq" podUID="03dfc5ea-523f-497c-a704-10c15bb768ba" containerName="manager" containerID="cri-o://656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e" gracePeriod=30
Sep 09 17:45:53.576054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.576024    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:45:53.667872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.667789    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-metrics-certs\") pod \"03dfc5ea-523f-497c-a704-10c15bb768ba\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") "
Sep 09 17:45:53.667872 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.667846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f5st\" (UniqueName: \"kubernetes.io/projected/03dfc5ea-523f-497c-a704-10c15bb768ba-kube-api-access-8f5st\") pod \"03dfc5ea-523f-497c-a704-10c15bb768ba\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") "
Sep 09 17:45:53.668128 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.667876    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-webhook-cert\") pod \"03dfc5ea-523f-497c-a704-10c15bb768ba\" (UID: \"03dfc5ea-523f-497c-a704-10c15bb768ba\") "
Sep 09 17:45:53.670324 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.670285    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03dfc5ea-523f-497c-a704-10c15bb768ba" (UID: "03dfc5ea-523f-497c-a704-10c15bb768ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:45:53.670469 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.670320    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "03dfc5ea-523f-497c-a704-10c15bb768ba" (UID: "03dfc5ea-523f-497c-a704-10c15bb768ba"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:45:53.670469 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.670337    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dfc5ea-523f-497c-a704-10c15bb768ba-kube-api-access-8f5st" (OuterVolumeSpecName: "kube-api-access-8f5st") pod "03dfc5ea-523f-497c-a704-10c15bb768ba" (UID: "03dfc5ea-523f-497c-a704-10c15bb768ba"). InnerVolumeSpecName "kube-api-access-8f5st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:45:53.769245 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.769189    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-metrics-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:53.769245 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.769237    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f5st\" (UniqueName: \"kubernetes.io/projected/03dfc5ea-523f-497c-a704-10c15bb768ba-kube-api-access-8f5st\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:53.769245 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:53.769247    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03dfc5ea-523f-497c-a704-10c15bb768ba-webhook-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:45:54.248605 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.248576    2558 generic.go:356] "Generic (PLEG): container finished" podID="03dfc5ea-523f-497c-a704-10c15bb768ba" containerID="656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e" exitCode=0
Sep 09 17:45:54.248838 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.248620    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-r87sq"
Sep 09 17:45:54.248838 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.248664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-r87sq" event={"ID":"03dfc5ea-523f-497c-a704-10c15bb768ba","Type":"ContainerDied","Data":"656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e"}
Sep 09 17:45:54.248838 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.248718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-r87sq" event={"ID":"03dfc5ea-523f-497c-a704-10c15bb768ba","Type":"ContainerDied","Data":"5755a23d6ed375107732e3bc1eeb3e4839e4ee161cfda9feff31aba4dfcf43e2"}
Sep 09 17:45:54.248838 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.248746    2558 scope.go:117] "RemoveContainer" containerID="656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e"
Sep 09 17:45:54.258266 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.258246    2558 scope.go:117] "RemoveContainer" containerID="656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e"
Sep 09 17:45:54.258540 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:45:54.258522    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e\": container with ID starting with 656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e not found: ID does not exist" containerID="656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e"
Sep 09 17:45:54.258588 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.258549    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e"} err="failed to get container status \"656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e\": rpc error: code = NotFound desc = could not find container \"656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e\": container with ID starting with 656eff159db2910a2d95b493ab89e6711304cff25d6f0dfb531d8de63c17963e not found: ID does not exist"
Sep 09 17:45:54.270296 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.270223    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-r87sq"]
Sep 09 17:45:54.272221 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.272201    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-r87sq"]
Sep 09 17:45:54.820822 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:45:54.820790    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03dfc5ea-523f-497c-a704-10c15bb768ba" path="/var/lib/kubelet/pods/03dfc5ea-523f-497c-a704-10c15bb768ba/volumes"
Sep 09 17:46:55.097747 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.097712    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5474dd89c7-g5psc"]
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098020    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03dfc5ea-523f-497c-a704-10c15bb768ba" containerName="manager"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098032    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="03dfc5ea-523f-497c-a704-10c15bb768ba" containerName="manager"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098044    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6299fa-a3cb-49e4-84d2-448abadf0784" containerName="payload-pre-processing"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098050    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6299fa-a3cb-49e4-84d2-448abadf0784" containerName="payload-pre-processing"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098061    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750ef4d6-026a-4b40-802c-5c64c5484f50" containerName="revoke-keys"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098066    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="750ef4d6-026a-4b40-802c-5c64c5484f50" containerName="revoke-keys"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098075    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e69cbb-2904-4ce0-a62b-ccac89a78922" containerName="payload-processing"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098080    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e69cbb-2904-4ce0-a62b-ccac89a78922" containerName="payload-processing"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098124    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e69cbb-2904-4ce0-a62b-ccac89a78922" containerName="payload-processing"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098130    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="750ef4d6-026a-4b40-802c-5c64c5484f50" containerName="revoke-keys"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098137    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="03dfc5ea-523f-497c-a704-10c15bb768ba" containerName="manager"
Sep 09 17:46:55.098228 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.098143    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6299fa-a3cb-49e4-84d2-448abadf0784" containerName="payload-pre-processing"
Sep 09 17:46:55.099994 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.099978    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.107216 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.107189    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 17:46:55.107429 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.107406    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 17:46:55.107595 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.107351    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 17:46:55.107780 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.107353    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fh5cx\""
Sep 09 17:46:55.110342 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.110318    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-g5psc"]
Sep 09 17:46:55.192209 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.192170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-webhook-cert\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.192409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.192221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-metrics-certs\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.192409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.192247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42qs8\" (UniqueName: \"kubernetes.io/projected/84f4ab09-e1dd-4718-a540-d35692c38825-kube-api-access-42qs8\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.292887 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.292852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42qs8\" (UniqueName: \"kubernetes.io/projected/84f4ab09-e1dd-4718-a540-d35692c38825-kube-api-access-42qs8\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.293099 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.292949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-webhook-cert\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.293099 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.293015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-metrics-certs\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.295482 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.295451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-webhook-cert\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.295482 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.295463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-metrics-certs\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.301458 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.301434    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42qs8\" (UniqueName: \"kubernetes.io/projected/84f4ab09-e1dd-4718-a540-d35692c38825-kube-api-access-42qs8\") pod \"maas-controller-5474dd89c7-g5psc\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") " pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.416281 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.416200    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:55.697087 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.697062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-g5psc"]
Sep 09 17:46:55.699141 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:46:55.699107    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f4ab09_e1dd_4718_a540_d35692c38825.slice/crio-4101c77e82fa44d7a473a09b7eb72321a1b83c2e42d9881c37028db475f6c2ec WatchSource:0}: Error finding container 4101c77e82fa44d7a473a09b7eb72321a1b83c2e42d9881c37028db475f6c2ec: Status 404 returned error can't find the container with id 4101c77e82fa44d7a473a09b7eb72321a1b83c2e42d9881c37028db475f6c2ec
Sep 09 17:46:55.700555 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:55.700536    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 17:46:56.584989 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:56.584887    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-g5psc" event={"ID":"84f4ab09-e1dd-4718-a540-d35692c38825","Type":"ContainerStarted","Data":"dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc"}
Sep 09 17:46:56.584989 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:56.584938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-g5psc" event={"ID":"84f4ab09-e1dd-4718-a540-d35692c38825","Type":"ContainerStarted","Data":"4101c77e82fa44d7a473a09b7eb72321a1b83c2e42d9881c37028db475f6c2ec"}
Sep 09 17:46:56.585403 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:56.585008    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:46:56.604062 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:46:56.604008    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5474dd89c7-g5psc" podStartSLOduration=1.024388425 podStartE2EDuration="1.603991127s" totalImagesPullingTime="579.602702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:46:56.602408466 +0000 UTC m=+1792.230561168" watchObservedRunningTime="2026-09-09 17:46:56.603991127 +0000 UTC m=+1792.232143850"
Sep 09 17:47:04.879577 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:04.879549    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:47:04.883300 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:04.883278    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:47:05.148779 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:05.148698    2558 scope.go:117] "RemoveContainer" containerID="9ababa3f6db2c26dab6c1e00376caf9d207f12d4a9807823cabcc9b98b961885"
Sep 09 17:47:05.157071 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:05.157049    2558 scope.go:117] "RemoveContainer" containerID="6fdb2cde7cd7aacb1b65c1b8b35cef4c2ce4dc050ffba75f00c0f98b2b9ff4df"
Sep 09 17:47:07.592799 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:07.592770    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:47:16.965422 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:16.965386    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-g5psc"]
Sep 09 17:47:16.965866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:16.965602    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5474dd89c7-g5psc" podUID="84f4ab09-e1dd-4718-a540-d35692c38825" containerName="manager" containerID="cri-o://dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc" gracePeriod=30
Sep 09 17:47:17.207402 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.207373    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:47:17.380852 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.380756    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-webhook-cert\") pod \"84f4ab09-e1dd-4718-a540-d35692c38825\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") "
Sep 09 17:47:17.381048 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.380868    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42qs8\" (UniqueName: \"kubernetes.io/projected/84f4ab09-e1dd-4718-a540-d35692c38825-kube-api-access-42qs8\") pod \"84f4ab09-e1dd-4718-a540-d35692c38825\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") "
Sep 09 17:47:17.381139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.381043    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-metrics-certs\") pod \"84f4ab09-e1dd-4718-a540-d35692c38825\" (UID: \"84f4ab09-e1dd-4718-a540-d35692c38825\") "
Sep 09 17:47:17.383084 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.383047    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84f4ab09-e1dd-4718-a540-d35692c38825" (UID: "84f4ab09-e1dd-4718-a540-d35692c38825"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:47:17.383084 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.383078    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "84f4ab09-e1dd-4718-a540-d35692c38825" (UID: "84f4ab09-e1dd-4718-a540-d35692c38825"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:47:17.383243 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.383214    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f4ab09-e1dd-4718-a540-d35692c38825-kube-api-access-42qs8" (OuterVolumeSpecName: "kube-api-access-42qs8") pod "84f4ab09-e1dd-4718-a540-d35692c38825" (UID: "84f4ab09-e1dd-4718-a540-d35692c38825"). InnerVolumeSpecName "kube-api-access-42qs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:47:17.482425 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.482383    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-webhook-cert\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:47:17.482425 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.482419    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42qs8\" (UniqueName: \"kubernetes.io/projected/84f4ab09-e1dd-4718-a540-d35692c38825-kube-api-access-42qs8\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:47:17.482425 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.482434    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f4ab09-e1dd-4718-a540-d35692c38825-metrics-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:47:17.693898 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.693858    2558 generic.go:356] "Generic (PLEG): container finished" podID="84f4ab09-e1dd-4718-a540-d35692c38825" containerID="dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc" exitCode=0
Sep 09 17:47:17.694080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.693907    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-g5psc"
Sep 09 17:47:17.694080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.693936    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-g5psc" event={"ID":"84f4ab09-e1dd-4718-a540-d35692c38825","Type":"ContainerDied","Data":"dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc"}
Sep 09 17:47:17.694080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.694009    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-g5psc" event={"ID":"84f4ab09-e1dd-4718-a540-d35692c38825","Type":"ContainerDied","Data":"4101c77e82fa44d7a473a09b7eb72321a1b83c2e42d9881c37028db475f6c2ec"}
Sep 09 17:47:17.694080 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.694032    2558 scope.go:117] "RemoveContainer" containerID="dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc"
Sep 09 17:47:17.702945 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.702925    2558 scope.go:117] "RemoveContainer" containerID="dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc"
Sep 09 17:47:17.703263 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:47:17.703243    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc\": container with ID starting with dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc not found: ID does not exist" containerID="dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc"
Sep 09 17:47:17.703325 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.703272    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc"} err="failed to get container status \"dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc\": rpc error: code = NotFound desc = could not find container \"dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc\": container with ID starting with dcca59c9d251a063a282e3af8fa0bd3a334fe29c5c2195cc8e0298286520f1cc not found: ID does not exist"
Sep 09 17:47:17.715818 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.715787    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-g5psc"]
Sep 09 17:47:17.718754 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:17.718729    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-g5psc"]
Sep 09 17:47:18.510298 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.506139    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5474dd89c7-z4z7q"]
Sep 09 17:47:18.510298 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.506801    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f4ab09-e1dd-4718-a540-d35692c38825" containerName="manager"
Sep 09 17:47:18.510298 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.506820    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f4ab09-e1dd-4718-a540-d35692c38825" containerName="manager"
Sep 09 17:47:18.510298 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.506991    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f4ab09-e1dd-4718-a540-d35692c38825" containerName="manager"
Sep 09 17:47:18.510298 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.510036    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.514122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.514095    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 17:47:18.514230 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.514177    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fh5cx\""
Sep 09 17:47:18.514230 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.514189    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 17:47:18.514318 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.514259    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 17:47:18.518973 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.518928    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-z4z7q"]
Sep 09 17:47:18.592122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.592076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9kf\" (UniqueName: \"kubernetes.io/projected/cfc8b893-0f97-4867-97d2-745c5ac3aab9-kube-api-access-vx9kf\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.592122 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.592124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfc8b893-0f97-4867-97d2-745c5ac3aab9-webhook-cert\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.592361 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.592187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc8b893-0f97-4867-97d2-745c5ac3aab9-metrics-certs\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.693694 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.693657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc8b893-0f97-4867-97d2-745c5ac3aab9-metrics-certs\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.693880 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.693764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9kf\" (UniqueName: \"kubernetes.io/projected/cfc8b893-0f97-4867-97d2-745c5ac3aab9-kube-api-access-vx9kf\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.693880 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.693796    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfc8b893-0f97-4867-97d2-745c5ac3aab9-webhook-cert\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.696395 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.696367    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfc8b893-0f97-4867-97d2-745c5ac3aab9-metrics-certs\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.696516 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.696419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfc8b893-0f97-4867-97d2-745c5ac3aab9-webhook-cert\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.703194 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.703169    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9kf\" (UniqueName: \"kubernetes.io/projected/cfc8b893-0f97-4867-97d2-745c5ac3aab9-kube-api-access-vx9kf\") pod \"maas-controller-5474dd89c7-z4z7q\" (UID: \"cfc8b893-0f97-4867-97d2-745c5ac3aab9\") " pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:18.816071 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.815991    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f4ab09-e1dd-4718-a540-d35692c38825" path="/var/lib/kubelet/pods/84f4ab09-e1dd-4718-a540-d35692c38825/volumes"
Sep 09 17:47:18.821294 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:18.821277    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:19.091050 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:19.091012    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5474dd89c7-z4z7q"]
Sep 09 17:47:19.092703 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:47:19.092676    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc8b893_0f97_4867_97d2_745c5ac3aab9.slice/crio-f62afecf40fb9f5ad4b59783364b64b458b8d2b55e6ecf824d6bbaf8a99b4328 WatchSource:0}: Error finding container f62afecf40fb9f5ad4b59783364b64b458b8d2b55e6ecf824d6bbaf8a99b4328: Status 404 returned error can't find the container with id f62afecf40fb9f5ad4b59783364b64b458b8d2b55e6ecf824d6bbaf8a99b4328
Sep 09 17:47:19.707509 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:19.707471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-z4z7q" event={"ID":"cfc8b893-0f97-4867-97d2-745c5ac3aab9","Type":"ContainerStarted","Data":"9a28b47814d64f59ec8de0e4dc68308dcbcd5ac122aab6d182b68f0ef1339e45"}
Sep 09 17:47:19.707934 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:19.707524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5474dd89c7-z4z7q" event={"ID":"cfc8b893-0f97-4867-97d2-745c5ac3aab9","Type":"ContainerStarted","Data":"f62afecf40fb9f5ad4b59783364b64b458b8d2b55e6ecf824d6bbaf8a99b4328"}
Sep 09 17:47:19.707934 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:19.707546    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:47:19.725834 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:19.725781    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5474dd89c7-z4z7q" podStartSLOduration=1.253107318 podStartE2EDuration="1.725765395s" totalImagesPullingTime="472.658077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:47:19.723636218 +0000 UTC m=+1815.351788921" watchObservedRunningTime="2026-09-09 17:47:19.725765395 +0000 UTC m=+1815.353918242"
Sep 09 17:47:30.715798 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:47:30.715763    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5474dd89c7-z4z7q"
Sep 09 17:48:05.179916 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:48:05.179887    2558 scope.go:117] "RemoveContainer" containerID="e5363a5f6006dcf146ba99b446a5baf0ba51c154c5f9d30ed26fd191a1892e5f"
Sep 09 17:50:00.781692 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.781606    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"]
Sep 09 17:50:00.784868 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.784849    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.787374 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.787348    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c2bfdbf6-kserve-self-signed-certs\""
Sep 09 17:50:00.792582 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.792563    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"]
Sep 09 17:50:00.800946 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.800684    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8f6\" (UniqueName: \"kubernetes.io/projected/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-kube-api-access-5v8f6\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.801164 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.801135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-home\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.801269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.801174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tmp-dir\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.801269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.801221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tls-certs\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.801269 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.801260    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-model-cache\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.801377 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.801284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-dshm\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902400 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-home\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902400 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902394    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tmp-dir\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902604 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tls-certs\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902654 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-model-cache\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902693 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-dshm\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902757 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8f6\" (UniqueName: \"kubernetes.io/projected/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-kube-api-access-5v8f6\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tmp-dir\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.902871 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.902802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-home\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.903054 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.903033    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-model-cache\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.904866 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.904842    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-dshm\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.905252 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.905231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tls-certs\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:00.910021 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:00.909997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8f6\" (UniqueName: \"kubernetes.io/projected/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-kube-api-access-5v8f6\") pod \"e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") " pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:01.094246 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:01.094163    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:01.367446 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:01.367418    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"]
Sep 09 17:50:01.369365 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:50:01.369337    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e8d0f4_25a9_4e51_a897_0fc53cc8f67a.slice/crio-c42e8845d1cd775713284c0173f34a9e1b6bbe15e29794faeefaa16d4348df51 WatchSource:0}: Error finding container c42e8845d1cd775713284c0173f34a9e1b6bbe15e29794faeefaa16d4348df51: Status 404 returned error can't find the container with id c42e8845d1cd775713284c0173f34a9e1b6bbe15e29794faeefaa16d4348df51
Sep 09 17:50:01.574467 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:01.574430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx" event={"ID":"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a","Type":"ContainerStarted","Data":"770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736"}
Sep 09 17:50:01.574467 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:01.574473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx" event={"ID":"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a","Type":"ContainerStarted","Data":"c42e8845d1cd775713284c0173f34a9e1b6bbe15e29794faeefaa16d4348df51"}
Sep 09 17:50:01.590577 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:01.590521    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx" podStartSLOduration=1.590503561 podStartE2EDuration="1.590503561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:50:01.589650376 +0000 UTC m=+1977.217803070" watchObservedRunningTime="2026-09-09 17:50:01.590503561 +0000 UTC m=+1977.218656303"
Sep 09 17:50:11.094591 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:11.094559    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:11.094985 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:11.094608    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:11.107279 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:11.107252    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:11.637735 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:11.637697    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:50:20.785763 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.785732    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"]
Sep 09 17:50:20.789113 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.789095    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.792008 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.791980    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c2bfdbf6-kserve-self-signed-certs\""
Sep 09 17:50:20.799198 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.799168    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"]
Sep 09 17:50:20.874456 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.874419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9pw\" (UniqueName: \"kubernetes.io/projected/6578261e-ce95-4c66-8325-015d961cec53-kube-api-access-bp9pw\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.874622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.874472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-home\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.874622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.874497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-dshm\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.874622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.874514    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6578261e-ce95-4c66-8325-015d961cec53-tls-certs\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.874622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.874539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-tmp-dir\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.874622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.874562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-model-cache\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975297 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9pw\" (UniqueName: \"kubernetes.io/projected/6578261e-ce95-4c66-8325-015d961cec53-kube-api-access-bp9pw\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-home\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-dshm\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6578261e-ce95-4c66-8325-015d961cec53-tls-certs\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975400    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-tmp-dir\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975489 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-model-cache\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975797 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-home\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975858 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975795    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-tmp-dir\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.975921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.975908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-model-cache\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.977801 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.977773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-dshm\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.978061 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.978045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6578261e-ce95-4c66-8325-015d961cec53-tls-certs\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:20.983662 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:20.983633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9pw\" (UniqueName: \"kubernetes.io/projected/6578261e-ce95-4c66-8325-015d961cec53-kube-api-access-bp9pw\") pod \"e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") " pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:21.099686 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:21.099591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:21.371437 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:21.371409    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"]
Sep 09 17:50:21.373411 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:50:21.373370    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6578261e_ce95_4c66_8325_015d961cec53.slice/crio-3c33351c194f9fc643769ad3c970bd184c01ccd7d48bfe78fff86d16a25b654d WatchSource:0}: Error finding container 3c33351c194f9fc643769ad3c970bd184c01ccd7d48bfe78fff86d16a25b654d: Status 404 returned error can't find the container with id 3c33351c194f9fc643769ad3c970bd184c01ccd7d48bfe78fff86d16a25b654d
Sep 09 17:50:21.678532 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:21.678497    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh" event={"ID":"6578261e-ce95-4c66-8325-015d961cec53","Type":"ContainerStarted","Data":"c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921"}
Sep 09 17:50:21.678720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:21.678539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh" event={"ID":"6578261e-ce95-4c66-8325-015d961cec53","Type":"ContainerStarted","Data":"3c33351c194f9fc643769ad3c970bd184c01ccd7d48bfe78fff86d16a25b654d"}
Sep 09 17:50:21.695132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:21.695086    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh" podStartSLOduration=1.695051458 podStartE2EDuration="1.695051458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:50:21.694258401 +0000 UTC m=+1997.322411103" watchObservedRunningTime="2026-09-09 17:50:21.695051458 +0000 UTC m=+1997.323204156"
Sep 09 17:50:31.100437 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:31.100385    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:31.100981 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:31.100538    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:31.113267 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:31.113230    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:31.744144 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:31.744110    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:50:46.583884 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:46.583847    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"]
Sep 09 17:50:48.198647 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:50:48.198603    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"]
Sep 09 17:51:01.604651 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.604588    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx" podUID="36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" containerName="main" containerID="cri-o://770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736" gracePeriod=45
Sep 09 17:51:01.626517 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.626480    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx" podUID="36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 09 17:51:01.849501 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.849475    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:51:01.892406 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.892307    2558 generic.go:356] "Generic (PLEG): container finished" podID="36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" containerID="770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736" exitCode=0
Sep 09 17:51:01.892406 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.892348    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx" event={"ID":"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a","Type":"ContainerDied","Data":"770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736"}
Sep 09 17:51:01.892406 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.892366    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"
Sep 09 17:51:01.892406 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.892383    2558 scope.go:117] "RemoveContainer" containerID="770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736"
Sep 09 17:51:01.892702 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.892374    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx" event={"ID":"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a","Type":"ContainerDied","Data":"c42e8845d1cd775713284c0173f34a9e1b6bbe15e29794faeefaa16d4348df51"}
Sep 09 17:51:01.901200 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.901175    2558 scope.go:117] "RemoveContainer" containerID="770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736"
Sep 09 17:51:01.901529 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:51:01.901505    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736\": container with ID starting with 770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736 not found: ID does not exist" containerID="770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736"
Sep 09 17:51:01.901574 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.901541    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736"} err="failed to get container status \"770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736\": rpc error: code = NotFound desc = could not find container \"770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736\": container with ID starting with 770a2b9c26224e65a0cdfc66a5fa2bcb835489454786165c4adb408e1d273736 not found: ID does not exist"
Sep 09 17:51:01.930014 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.929983    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-model-cache\") pod \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") "
Sep 09 17:51:01.930171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930030    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-dshm\") pod \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") "
Sep 09 17:51:01.930171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930050    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-home\") pod \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") "
Sep 09 17:51:01.930171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tmp-dir\") pod \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") "
Sep 09 17:51:01.930171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930085    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tls-certs\") pod \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") "
Sep 09 17:51:01.930171 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930115    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v8f6\" (UniqueName: \"kubernetes.io/projected/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-kube-api-access-5v8f6\") pod \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\" (UID: \"36e8d0f4-25a9-4e51-a897-0fc53cc8f67a\") "
Sep 09 17:51:01.930432 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930304    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-model-cache" (OuterVolumeSpecName: "model-cache") pod "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" (UID: "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:01.930432 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930312    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-home" (OuterVolumeSpecName: "home") pod "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" (UID: "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:01.930432 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.930387    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" (UID: "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:01.932852 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.932558    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" (UID: "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:51:01.932993 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.932849    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-dshm" (OuterVolumeSpecName: "dshm") pod "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" (UID: "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:01.932993 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:01.932672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-kube-api-access-5v8f6" (OuterVolumeSpecName: "kube-api-access-5v8f6") pod "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" (UID: "36e8d0f4-25a9-4e51-a897-0fc53cc8f67a"). InnerVolumeSpecName "kube-api-access-5v8f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:51:02.030720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.030674    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:02.030720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.030711    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:02.030939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.030747    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:02.030939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.030761    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:02.030939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.030773    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8f6\" (UniqueName: \"kubernetes.io/projected/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-kube-api-access-5v8f6\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:02.030939 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.030785    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:02.213944 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.213907    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"]
Sep 09 17:51:02.217218 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.217188    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c2bfdbf6-kserve-8669f7f97f-6vlnx"]
Sep 09 17:51:02.816172 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:02.816137    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" path="/var/lib/kubelet/pods/36e8d0f4-25a9-4e51-a897-0fc53cc8f67a/volumes"
Sep 09 17:51:03.218786 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.218718    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh" podUID="6578261e-ce95-4c66-8325-015d961cec53" containerName="main" containerID="cri-o://c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921" gracePeriod=45
Sep 09 17:51:03.478690 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.478667    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:51:03.543299 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543259    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp9pw\" (UniqueName: \"kubernetes.io/projected/6578261e-ce95-4c66-8325-015d961cec53-kube-api-access-bp9pw\") pod \"6578261e-ce95-4c66-8325-015d961cec53\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") "
Sep 09 17:51:03.543299 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543306    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-model-cache\") pod \"6578261e-ce95-4c66-8325-015d961cec53\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") "
Sep 09 17:51:03.543511 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543329    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-tmp-dir\") pod \"6578261e-ce95-4c66-8325-015d961cec53\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") "
Sep 09 17:51:03.543511 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543363    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6578261e-ce95-4c66-8325-015d961cec53-tls-certs\") pod \"6578261e-ce95-4c66-8325-015d961cec53\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") "
Sep 09 17:51:03.543591 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-dshm\") pod \"6578261e-ce95-4c66-8325-015d961cec53\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") "
Sep 09 17:51:03.543641 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543613    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-model-cache" (OuterVolumeSpecName: "model-cache") pod "6578261e-ce95-4c66-8325-015d961cec53" (UID: "6578261e-ce95-4c66-8325-015d961cec53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:03.543683 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543621    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6578261e-ce95-4c66-8325-015d961cec53" (UID: "6578261e-ce95-4c66-8325-015d961cec53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:03.543683 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543657    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-home\") pod \"6578261e-ce95-4c66-8325-015d961cec53\" (UID: \"6578261e-ce95-4c66-8325-015d961cec53\") "
Sep 09 17:51:03.543876 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543842    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-home" (OuterVolumeSpecName: "home") pod "6578261e-ce95-4c66-8325-015d961cec53" (UID: "6578261e-ce95-4c66-8325-015d961cec53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:03.543936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543855    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-model-cache\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:03.543936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.543907    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-tmp-dir\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:03.545588 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.545562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6578261e-ce95-4c66-8325-015d961cec53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6578261e-ce95-4c66-8325-015d961cec53" (UID: "6578261e-ce95-4c66-8325-015d961cec53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 17:51:03.545698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.545557    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-dshm" (OuterVolumeSpecName: "dshm") pod "6578261e-ce95-4c66-8325-015d961cec53" (UID: "6578261e-ce95-4c66-8325-015d961cec53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 17:51:03.545698 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.545589    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6578261e-ce95-4c66-8325-015d961cec53-kube-api-access-bp9pw" (OuterVolumeSpecName: "kube-api-access-bp9pw") pod "6578261e-ce95-4c66-8325-015d961cec53" (UID: "6578261e-ce95-4c66-8325-015d961cec53"). InnerVolumeSpecName "kube-api-access-bp9pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 17:51:03.644774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.644731    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6578261e-ce95-4c66-8325-015d961cec53-tls-certs\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:03.644774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.644767    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-dshm\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:03.644774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.644775    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6578261e-ce95-4c66-8325-015d961cec53-home\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:03.644774 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.644783    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9pw\" (UniqueName: \"kubernetes.io/projected/6578261e-ce95-4c66-8325-015d961cec53-kube-api-access-bp9pw\") on node \"ip-10-0-139-100.ec2.internal\" DevicePath \"\""
Sep 09 17:51:03.905599 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.905503    2558 generic.go:356] "Generic (PLEG): container finished" podID="6578261e-ce95-4c66-8325-015d961cec53" containerID="c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921" exitCode=0
Sep 09 17:51:03.905599 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.905556    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"
Sep 09 17:51:03.906119 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.905592    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh" event={"ID":"6578261e-ce95-4c66-8325-015d961cec53","Type":"ContainerDied","Data":"c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921"}
Sep 09 17:51:03.906119 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.905643    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh" event={"ID":"6578261e-ce95-4c66-8325-015d961cec53","Type":"ContainerDied","Data":"3c33351c194f9fc643769ad3c970bd184c01ccd7d48bfe78fff86d16a25b654d"}
Sep 09 17:51:03.906119 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.905665    2558 scope.go:117] "RemoveContainer" containerID="c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921"
Sep 09 17:51:03.915844 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.915823    2558 scope.go:117] "RemoveContainer" containerID="c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921"
Sep 09 17:51:03.916184 ip-10-0-139-100 kubenswrapper[2558]: E0909 17:51:03.916163    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921\": container with ID starting with c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921 not found: ID does not exist" containerID="c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921"
Sep 09 17:51:03.916268 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.916192    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921"} err="failed to get container status \"c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921\": rpc error: code = NotFound desc = could not find container \"c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921\": container with ID starting with c3839b54656d158efbfd9111a2b23e280425b7fd18ca083933fea88cd1545921 not found: ID does not exist"
Sep 09 17:51:03.927625 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.927599    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"]
Sep 09 17:51:03.934332 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:03.934307    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c2bfdbf6-kserve-674ddf8d87-9vbjh"]
Sep 09 17:51:04.816584 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:04.816552    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6578261e-ce95-4c66-8325-015d961cec53" path="/var/lib/kubelet/pods/6578261e-ce95-4c66-8325-015d961cec53/volumes"
Sep 09 17:51:05.221212 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:05.221185    2558 scope.go:117] "RemoveContainer" containerID="02480ed8a5500c4696a641c040665ebdd9210916fce44fc7dd698e76af955db6"
Sep 09 17:51:05.229400 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:05.229380    2558 scope.go:117] "RemoveContainer" containerID="1ad4efc9732e5cc5e5dd244508a241447f68088b328cda5a9124baeedc0ec4b9"
Sep 09 17:51:05.236936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:05.236919    2558 scope.go:117] "RemoveContainer" containerID="f5756752c5078711f9ea8a69ce905949642698ba6d8d34d59565f98f545454be"
Sep 09 17:51:05.244033 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:05.244016    2558 scope.go:117] "RemoveContainer" containerID="4a477dc38001bbe20f9faf8d7cc06b9b7301a9bde35daba41146d746ce8cbcdb"
Sep 09 17:51:41.736381 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:41.736304    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8db654b6d-zpcnv_62c85067-f231-4236-9c39-ce99c514a88d/authorino/0.log"
Sep 09 17:51:47.521355 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:47.521325    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5474dd89c7-z4z7q_cfc8b893-0f97-4867-97d2-745c5ac3aab9/manager/0.log"
Sep 09 17:51:47.903525 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:47.903447    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-7dhs9_9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307/copy-manifests/0.log"
Sep 09 17:51:47.910136 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:47.910106    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-7dhs9_9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307/manager/0.log"
Sep 09 17:51:48.878610 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:48.878580    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-sl7kq_2ca2bd2a-96d9-4cc2-bf87-006af20f2cfe/postgres/0.log"
Sep 09 17:51:49.683110 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:49.683078    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8db654b6d-zpcnv_62c85067-f231-4236-9c39-ce99c514a88d/authorino/0.log"
Sep 09 17:51:51.172740 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:51.172708    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-t9l6t_3dff361f-6c81-4d16-9ff2-e81651a5d04a/kube-auth-proxy/0.log"
Sep 09 17:51:51.440772 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:51.440742    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4gqc5_22e1e33b-0d25-4a1e-8aa2-dce604fda858/payload-pre-processing/0.log"
Sep 09 17:51:51.557009 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:51.556976    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95zgv_d2f6aefe-ce42-414c-a1ce-fc534c57a331/payload-processing/0.log"
Sep 09 17:51:52.482498 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:52.482469    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f77gkq_1e724880-5662-4425-899e-4e9d2e69615a/main/0.log"
Sep 09 17:51:52.599384 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:52.599353    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hwrdx_ad773972-ff4a-4f42-a60d-115371b3349d/main/0.log"
Sep 09 17:51:52.721419 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:52.721393    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-z6rkn_ea1f7675-1366-4c07-976e-18267b0542c9/main/0.log"
Sep 09 17:51:58.373923 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.373883    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8k64x/must-gather-vt77g"]
Sep 09 17:51:58.374387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.374272    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6578261e-ce95-4c66-8325-015d961cec53" containerName="main"
Sep 09 17:51:58.374387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.374289    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6578261e-ce95-4c66-8325-015d961cec53" containerName="main"
Sep 09 17:51:58.374387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.374303    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" containerName="main"
Sep 09 17:51:58.374387 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.374312    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" containerName="main"
Sep 09 17:51:58.374611 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.374466    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="36e8d0f4-25a9-4e51-a897-0fc53cc8f67a" containerName="main"
Sep 09 17:51:58.374611 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.374481    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6578261e-ce95-4c66-8325-015d961cec53" containerName="main"
Sep 09 17:51:58.377836 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.377808    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.380469 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.380448    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k64x\"/\"kube-root-ca.crt\""
Sep 09 17:51:58.382596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.382565    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k64x\"/\"openshift-service-ca.crt\""
Sep 09 17:51:58.382705 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.382609    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8k64x\"/\"default-dockercfg-f5w8x\""
Sep 09 17:51:58.385154 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.384885    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k64x/must-gather-vt77g"]
Sep 09 17:51:58.411813 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.411775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5p7n\" (UniqueName: \"kubernetes.io/projected/6e5012a2-6345-4201-a135-495f5e8c2508-kube-api-access-f5p7n\") pod \"must-gather-vt77g\" (UID: \"6e5012a2-6345-4201-a135-495f5e8c2508\") " pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.411993 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.411816    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6e5012a2-6345-4201-a135-495f5e8c2508-must-gather-output\") pod \"must-gather-vt77g\" (UID: \"6e5012a2-6345-4201-a135-495f5e8c2508\") " pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.512746 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.512698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5p7n\" (UniqueName: \"kubernetes.io/projected/6e5012a2-6345-4201-a135-495f5e8c2508-kube-api-access-f5p7n\") pod \"must-gather-vt77g\" (UID: \"6e5012a2-6345-4201-a135-495f5e8c2508\") " pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.512947 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.512757    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6e5012a2-6345-4201-a135-495f5e8c2508-must-gather-output\") pod \"must-gather-vt77g\" (UID: \"6e5012a2-6345-4201-a135-495f5e8c2508\") " pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.513177 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.513155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6e5012a2-6345-4201-a135-495f5e8c2508-must-gather-output\") pod \"must-gather-vt77g\" (UID: \"6e5012a2-6345-4201-a135-495f5e8c2508\") " pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.520668 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.520647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5p7n\" (UniqueName: \"kubernetes.io/projected/6e5012a2-6345-4201-a135-495f5e8c2508-kube-api-access-f5p7n\") pod \"must-gather-vt77g\" (UID: \"6e5012a2-6345-4201-a135-495f5e8c2508\") " pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.687471 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.687412    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k64x/must-gather-vt77g"
Sep 09 17:51:58.957905 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.957877    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k64x/must-gather-vt77g"]
Sep 09 17:51:58.960069 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:51:58.960036    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e5012a2_6345_4201_a135_495f5e8c2508.slice/crio-bebde417ec932cb4814d1ea11bf72a27115af0a0f0f5ccc7abf2bed5f662b385 WatchSource:0}: Error finding container bebde417ec932cb4814d1ea11bf72a27115af0a0f0f5ccc7abf2bed5f662b385: Status 404 returned error can't find the container with id bebde417ec932cb4814d1ea11bf72a27115af0a0f0f5ccc7abf2bed5f662b385
Sep 09 17:51:58.962284 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:58.962268    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 17:51:59.179622 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:51:59.179589    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k64x/must-gather-vt77g" event={"ID":"6e5012a2-6345-4201-a135-495f5e8c2508","Type":"ContainerStarted","Data":"bebde417ec932cb4814d1ea11bf72a27115af0a0f0f5ccc7abf2bed5f662b385"}
Sep 09 17:52:01.192905 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:01.192868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k64x/must-gather-vt77g" event={"ID":"6e5012a2-6345-4201-a135-495f5e8c2508","Type":"ContainerStarted","Data":"571698bc198cc82d917afff24f5f2fa0ff4ff40cb31aee5735baf8c83089f044"}
Sep 09 17:52:02.202748 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:02.202692    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k64x/must-gather-vt77g" event={"ID":"6e5012a2-6345-4201-a135-495f5e8c2508","Type":"ContainerStarted","Data":"a149e88b4a946eb4a781fb8e13556633feeb2bc2b13115b326e2faf5da9e17ff"}
Sep 09 17:52:02.219450 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:02.219392    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8k64x/must-gather-vt77g" podStartSLOduration=2.1761760900000002 podStartE2EDuration="4.219376073s" totalImagesPullingTime="2.043199983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:52:02.21730542 +0000 UTC m=+2097.845458125" watchObservedRunningTime="2026-09-09 17:52:02.219376073 +0000 UTC m=+2097.847528776"
Sep 09 17:52:02.484897 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:02.484820    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l8zcg_56c38f48-b836-41fb-92a1-2190725d2d18/global-pull-secret-syncer/0.log"
Sep 09 17:52:02.647820 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:02.647792    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qngb4_19a374f5-cf67-4db9-bd80-518a808fcd26/konnectivity-agent/0.log"
Sep 09 17:52:02.741981 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:02.741880    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-100.ec2.internal_a63bc5640b9cbe7de0e68e356f93dd93/haproxy/0.log"
Sep 09 17:52:04.973137 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:04.973104    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:52:04.975720 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:04.975687    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:52:05.284878 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:05.284779    2558 scope.go:117] "RemoveContainer" containerID="17c7c54b45de10e2c8d16ee29dd2854394894560655f756679fa96248b6164d2"
Sep 09 17:52:05.308295 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:05.308243    2558 scope.go:117] "RemoveContainer" containerID="a2b9eb0fd37ef6e20b739d1756cfe81c32a2db48f6459bd1ef8d0d23601aeff6"
Sep 09 17:52:07.825976 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:07.825671    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8db654b6d-zpcnv_62c85067-f231-4236-9c39-ce99c514a88d/authorino/0.log"
Sep 09 17:52:09.798668 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:09.798582    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kp2k_b1405646-f328-4dd5-8515-413e560f5748/node-exporter/0.log"
Sep 09 17:52:09.828555 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:09.828518    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kp2k_b1405646-f328-4dd5-8515-413e560f5748/kube-rbac-proxy/0.log"
Sep 09 17:52:09.849334 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:09.849300    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kp2k_b1405646-f328-4dd5-8515-413e560f5748/init-textfile/0.log"
Sep 09 17:52:10.234390 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:10.234356    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2c9fz_fec38056-a2ad-4785-ac80-69dc61eb7d17/prometheus-operator/0.log"
Sep 09 17:52:10.251947 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:10.251903    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2c9fz_fec38056-a2ad-4785-ac80-69dc61eb7d17/kube-rbac-proxy/0.log"
Sep 09 17:52:13.017043 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.017015    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65f65c7f66-hfmc9_27d7169e-87bf-47cb-abc2-ea4bc55c4ff7/console/0.log"
Sep 09 17:52:13.221936 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.221897    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"]
Sep 09 17:52:13.228700 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.228666    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.235689 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.235653    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"]
Sep 09 17:52:13.360457 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.360375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-proc\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.360457 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.360446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-lib-modules\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.360686 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.360482    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfjf\" (UniqueName: \"kubernetes.io/projected/03a83a41-6c97-4c44-858e-2b1f28ab9b85-kube-api-access-xsfjf\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.360686 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.360502    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-sys\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.360686 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.360568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-podres\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.461734 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.461690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-podres\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.461743    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-proc\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.461805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-lib-modules\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.461844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfjf\" (UniqueName: \"kubernetes.io/projected/03a83a41-6c97-4c44-858e-2b1f28ab9b85-kube-api-access-xsfjf\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.461872    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-sys\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.461882    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-podres\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.461887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-proc\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462132 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.462029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-sys\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.462537 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.462158    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03a83a41-6c97-4c44-858e-2b1f28ab9b85-lib-modules\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.470255 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.470225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfjf\" (UniqueName: \"kubernetes.io/projected/03a83a41-6c97-4c44-858e-2b1f28ab9b85-kube-api-access-xsfjf\") pod \"perf-node-gather-daemonset-brhp4\" (UID: \"03a83a41-6c97-4c44-858e-2b1f28ab9b85\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.543191 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.542900    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:13.835514 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:13.835035    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"]
Sep 09 17:52:13.837434 ip-10-0-139-100 kubenswrapper[2558]: W0909 17:52:13.837392    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod03a83a41_6c97_4c44_858e_2b1f28ab9b85.slice/crio-f8888760ef1df82f5d73438687207dd10828b6fe8b81bf17e29aa2457383e2bb WatchSource:0}: Error finding container f8888760ef1df82f5d73438687207dd10828b6fe8b81bf17e29aa2457383e2bb: Status 404 returned error can't find the container with id f8888760ef1df82f5d73438687207dd10828b6fe8b81bf17e29aa2457383e2bb
Sep 09 17:52:14.308887 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:14.308843    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4" event={"ID":"03a83a41-6c97-4c44-858e-2b1f28ab9b85","Type":"ContainerStarted","Data":"359d503012c2cefa3774e97fabb7ba9592be330498171c026d65e1a60e999bd3"}
Sep 09 17:52:14.309376 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:14.308916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4" event={"ID":"03a83a41-6c97-4c44-858e-2b1f28ab9b85","Type":"ContainerStarted","Data":"f8888760ef1df82f5d73438687207dd10828b6fe8b81bf17e29aa2457383e2bb"}
Sep 09 17:52:14.309732 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:14.309707    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:14.327082 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:14.326997    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4" podStartSLOduration=1.326979733 podStartE2EDuration="1.326979733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:52:14.326191312 +0000 UTC m=+2109.954344015" watchObservedRunningTime="2026-09-09 17:52:14.326979733 +0000 UTC m=+2109.955132445"
Sep 09 17:52:14.966314 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:14.966269    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hcclw_ef37b751-5814-4867-930a-9632adbcc0c0/dns/0.log"
Sep 09 17:52:14.984972 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:14.984919    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hcclw_ef37b751-5814-4867-930a-9632adbcc0c0/kube-rbac-proxy/0.log"
Sep 09 17:52:15.091127 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:15.091086    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pdwrx_849b01c7-caef-49a5-b7e2-b2274e4ff53b/dns-node-resolver/0.log"
Sep 09 17:52:15.627814 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:15.627782    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qpr8j_b07f3322-92ad-4575-af6a-ae77de513515/node-ca/0.log"
Sep 09 17:52:16.928533 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:16.928503    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-t9l6t_3dff361f-6c81-4d16-9ff2-e81651a5d04a/kube-auth-proxy/0.log"
Sep 09 17:52:17.009454 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:17.009424    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4gqc5_22e1e33b-0d25-4a1e-8aa2-dce604fda858/payload-pre-processing/0.log"
Sep 09 17:52:17.048809 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:17.048764    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95zgv_d2f6aefe-ce42-414c-a1ce-fc534c57a331/payload-processing/0.log"
Sep 09 17:52:17.905341 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:17.905308    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qdcbd_c13bf76c-fc53-4e46-a5ce-4a4b5e37c867/serve-healthcheck-canary/0.log"
Sep 09 17:52:18.378812 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:18.378779    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5h2t_d03fbd8e-7c13-46fc-a575-812ba078b61d/kube-rbac-proxy/0.log"
Sep 09 17:52:18.400949 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:18.400923    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5h2t_d03fbd8e-7c13-46fc-a575-812ba078b61d/exporter/0.log"
Sep 09 17:52:18.420921 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:18.420896    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5h2t_d03fbd8e-7c13-46fc-a575-812ba078b61d/extractor/0.log"
Sep 09 17:52:20.811596 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:20.811539    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5474dd89c7-z4z7q_cfc8b893-0f97-4867-97d2-745c5ac3aab9/manager/0.log"
Sep 09 17:52:20.970299 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:20.970268    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-7dhs9_9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307/manager/0.log"
Sep 09 17:52:20.991327 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:20.991298    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-7dhs9_9b1ce6fb-a8a9-4eb3-b9fc-c5a7d51f2307/copy-manifests/0.log"
Sep 09 17:52:21.334253 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:21.334222    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-brhp4"
Sep 09 17:52:22.260462 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:22.260434    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67d8946886-rmz2c_b01bb6bd-0677-4010-9acd-3aa55a7d75dc/manager/0.log"
Sep 09 17:52:22.284522 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:22.284497    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zswbz_c21d2dfd-0f4b-4cc0-82fa-67a5f107ba18/openshift-lws-operator/0.log"
Sep 09 17:52:23.510827 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:23.510796    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-t9l6t_3dff361f-6c81-4d16-9ff2-e81651a5d04a/kube-auth-proxy/0.log"
Sep 09 17:52:23.574533 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:23.574444    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4gqc5_22e1e33b-0d25-4a1e-8aa2-dce604fda858/payload-pre-processing/0.log"
Sep 09 17:52:23.607584 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:23.607471    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95zgv_d2f6aefe-ce42-414c-a1ce-fc534c57a331/payload-processing/0.log"
Sep 09 17:52:25.520214 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:25.520186    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-t9l6t_3dff361f-6c81-4d16-9ff2-e81651a5d04a/kube-auth-proxy/0.log"
Sep 09 17:52:25.583907 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:25.583876    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4gqc5_22e1e33b-0d25-4a1e-8aa2-dce604fda858/payload-pre-processing/0.log"
Sep 09 17:52:25.619845 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:25.619760    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-95zgv_d2f6aefe-ce42-414c-a1ce-fc534c57a331/payload-processing/0.log"
Sep 09 17:52:29.142697 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.142668    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptt54_cf0778c3-c4ce-42d6-b7e0-51544b593949/kube-multus-additional-cni-plugins/0.log"
Sep 09 17:52:29.162189 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.162163    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptt54_cf0778c3-c4ce-42d6-b7e0-51544b593949/egress-router-binary-copy/0.log"
Sep 09 17:52:29.181439 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.181416    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptt54_cf0778c3-c4ce-42d6-b7e0-51544b593949/cni-plugins/0.log"
Sep 09 17:52:29.201485 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.201450    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptt54_cf0778c3-c4ce-42d6-b7e0-51544b593949/bond-cni-plugin/0.log"
Sep 09 17:52:29.220139 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.220113    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptt54_cf0778c3-c4ce-42d6-b7e0-51544b593949/routeoverride-cni/0.log"
Sep 09 17:52:29.239220 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.239196    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptt54_cf0778c3-c4ce-42d6-b7e0-51544b593949/whereabouts-cni-bincopy/0.log"
Sep 09 17:52:29.257894 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.257871    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptt54_cf0778c3-c4ce-42d6-b7e0-51544b593949/whereabouts-cni/0.log"
Sep 09 17:52:29.307372 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.307342    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c84jg_5a428b72-d41b-4a99-a171-75bfccbbe1b9/kube-multus/0.log"
Sep 09 17:52:29.446530 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.446505    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zmm4h_92691a45-2af1-48ac-830a-ead94a5117e6/network-metrics-daemon/0.log"
Sep 09 17:52:29.461754 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:29.461731    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zmm4h_92691a45-2af1-48ac-830a-ead94a5117e6/kube-rbac-proxy/0.log"
Sep 09 17:52:30.255743 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.255716    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-controller/0.log"
Sep 09 17:52:30.272833 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.272808    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/0.log"
Sep 09 17:52:30.273022 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.272844    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovn-acl-logging/1.log"
Sep 09 17:52:30.291860 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.291833    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/kube-rbac-proxy-node/0.log"
Sep 09 17:52:30.311799 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.311774    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 17:52:30.326831 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.326810    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/northd/0.log"
Sep 09 17:52:30.347409 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.347389    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/nbdb/0.log"
Sep 09 17:52:30.366688 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.366663    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/sbdb/0.log"
Sep 09 17:52:30.422794 ip-10-0-139-100 kubenswrapper[2558]: I0909 17:52:30.422762    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-snbz8_589eb0f8-9a41-4c82-bb43-e7b0e1e4a2b8/ovnkube-controller/0.log"